Keynote: David Beazley

all right can everybody hear me okay just before I begin I just have really honored to give the talk I'm you know the the plenary talk Dave Beasley I'm actually stunned to see so many people out there I was actually chair of the Python conference in 1998 when I had about 200 people and it just about killed me so I would actually like to have everybody applaud Jesse dollar again just I'm so so so in doing this talk on I'm sort of asked by the committee why don't you come talk about something diabolical so I had this on the slide here and I gave that some thought it's like diabolical sort of thinking wait what do I get out of coming to PyCon and for me the best part of Python is learning about new stuff and so I actually decided to do this talk on a topic of which I knew nothing about prior to preparing it and so let's talk about pi PI essentially pi PI if you don't know is Python implemented in Python which at first glance if you describe this to somebody they'll think you're insane right it's like Python implemented in Python it is quite a bit faster than normal Python through various magic and just as an example of that here is sort of some sample code that you can ride has anybody actually seen this Mandelbrot set code on the web I don't know whether Jeff is here or not but if he is he should come talk to me afterwards will I will sort you out but this is the some code that renders a Mandelbrot from Mandelbrot set you run that in C Python takes about 500 seconds you run it in pi PI and it's like 200 seconds doing nothing you know just like a two and a half times speed-up pretty amazing but this really doesn't even do pi PI justice actually I mean I have some Mandelbrot set code of my own that I use sometimes in in training classes and I try that I get it running 34 times faster I mean so I have the non off you skated you know Python code I think that's just amazing and the thing I really like about this is that I don't have to do anything I mean I basically just download you know pi PI and run it this solves my attempt to subvert the pike on Khan with uh with uh with parole program either but you know there's something there in pipe I was like wow this is great I don't have to do see extensions I don't have to like mess around and stuff this runs faster so so you can look at that project and you can say okay you know it's clearly seems to be faster than cpython in many ways there's a lot of magic going on underneath the covers of bye-bye it's doing just-in-time compilation it doing translation to see a lot of fairy van't advanced optimization going on but that's actually not what i want to talk about in this talk so this is not really a performance talk what I really want to talk about is which one can you mess around with and maybe fix with a pocket knife or something like that maybe in the dark or under a deadline or you know you know I actually have some experience with actually a little bit Curatola w bug okay I have a few did did any of you have to fix it but like a screwdriver or duct tape or anything like yeah yeah okay so this is something that I actually have some experience with and I would just like to comment for those of you at the tip boss last night I hope you left the pizza guy - okay Catan I drove a bug delivering pizza before I was doing programming and I did have to actually fix it with a pocket knife from time to time so so so what this talk is really um really bad it's just this idea of thinking about tinkering with pie pie is the pie pie project with all of that performance is that's something that you can mess around with and if so how do you do it so and I think tinkering is something that's actually really important if you think about cpython you've got all the patches that people apply to it you have the different tips that people propose you have extensions to it and and and my my favorite you're gonna love this you even have Python ideas okay so so there's you know there's a whole ecosystem of hacking on see Python and to me it really matters I mean you know a number of you saw you know like talks that I've given about the global interpreter lock I wouldn't have been able to do that if I couldn't tinker with it I mean if it was you know that wasn't possible if I go further back in my history I mean one of the first things I did with Python was ported to the connection machine which is a big parallel computer and you know in doing that we we didn't look at Python for performance we actually looked at it because we could hack on it I mean that was it was like simple enough of an implementation that you could hack on it and do stuff with it and just at the conference here I think this tinkering stuff just makes cool stuff I've been just been blown away by the sort of the ipython notebook stuff that I've been seeing and I know there's like a ton of tinkering and nothing around went into that and so so this question about pipe i I mean if this you know if pipe I has the promise it always are great performance and I can be totally lazy and I don't have to think about anything run around you know just just run it I have this question you know it's is it something that you can mess around with and how do you do it and I worry a little bit that it might just be this evil genius project I mean I actually worry that the programming model for pipeline might be that this from that this is from the tutorial the other day we have you know we have actually mike mueller up front with alex magic and arm in surrounding and those are the pipe ah some of them you know the pipe i core developers you know doing something and so that's that's actually something to think about and just us a little confession i actually will admit that pipe i scares me a little bit i don't know whether it scares anybody else but it does scare me a little bit just because of the you know the lot of moving parts a lot of pieces to it i worry a lot about the complexity of it you know is it is it like the you know the super board from last year or is it like a Cray one which is the second go at the same time period which is over on the right so this this question of you know can you mess around with pipe I and it's an honest question to ask and the answers I don't know I'm just going to admit the okay that's the punchline of this talk I don't know whether you can mess around with it or not but I decided to an experiment with that so that's that's really what I'm going to talk about is that's just this experiment can you go in and mess around with pipe I doing it from scratch I have actually never done anything with the pipe I project before never contributed any patch to it I submitted like one bug report against the global interpreter lock because that's my duty but that's like the the extent of it and I sort of wanted to see whether I could do this just using what they provide like source code documentation papers and so forth there are some constraints I do have kids married I'm trying to sell condos I have to run away from Julie's steal all the time because I'm like so far behind on the on the Python cookbook project you know there's like like other things going on so this had to be sort of a part-time project and then just so you know I'm tinkering with pipe I to me it's not the same as just using it I mean if I'm going to use it to download it and use it that's not very interesting and the other thing is that tinkering with it it's not the same as creating it I don't have any illusion that I'm just going to download this and implement a JIT right I mean it's you know I'm not smart enough to do that so what I'm thinking of is more things like bug reports maybe make extensions to it maybe patches study it you guys need to start pipe ideas by the way so post on there but but that's really what I'm thinking about so what I'm going to do is actually kind of take you through this sort of you know what it looks like to mess with this and if you you know I would actually encourage you to kind of to follow along although maybe I don't want all 1,800 people downloading it right this second off the off the webpage but if you want to start you know go to pi pi org and to mess with it you really do need the source code you know that's going to make me a source code as for tinkering so go get that one of the things that you'll find is that pi pi is indeed a version of python written in python this is this this sort of blows your mind the first time you sort of see this but if you if you go download the the source for pi PI you'll get you know maybe like a pipe i18 directory and if you go in there they're sort of a bin directory that you can find and there's a script in there called PI dot bi and you run it under Python I'm actually running it under Python two seven here so you can you can type that and it will sit there and kind of think about it for a little bit it's a little bit frightening it's running the C compiler at the moment talk about that later but it's sitting there very slowly coming up it's actually making java look kind of good right now so wait just a second here for the okay eighteen seconds to start up okay so but it is indeed a version of Python written in Python okay so you can type Python code into it now you probably would not want to use it like that I mean the performance of that is terrible I think Alex Gaynor at one point told me it's like what like 2,000 times slower than Python or something so you wouldn't do that and it turns out that that what you really do with PI Phi is you have to translate it into something else and this is where like your head will start to spin a little bit if you look at this project essentially what they tell you to do is that if you want the really high-performance version of pi pi you've got to go through this translation step so what that looks like on let's let's let's get out of that you essentially tell you to go down into a deep subdirectory of the project and then you just run translate with some options and then you just sit back okay so now this actually takes a while and it talked about what hap what happens here but this is essentially building pi PI from source actually a little cured unless build pi PI from source ok fair fair number of you since I'm going to live dangerously I'm going to leave that running on my machine on battery power it will serve as a as a natural cutoff for the end of the talk probably so so you can so you can build that and actually one of the the sort of the most formidable or formidable things about pie pie at least to me is an outsider it's actually this build process this is a this is a little movie of what it does running significantly faster okay so if you start building pie pie it starts going through Mandelbrot sets and all these things I'm starting to get terrified because I already hate make files and you know they might as well just put like a keep out sign I think go away but what is actually happening here is pretty amazing okay well first first all facts this movie is running 64 times faster than normal actually takes a few hours to do this just in contrast building cpython takes about 90 seconds on this machine so it's significantly longer build time what is actually doing oh actually one thing before you get to that one immediate problem it's actually fine enough ram to do this I only actually have two machines where I can build this thing one is my laptop with 8 gig of ram and the other is an amazon to extra-large instance with 17 gigabytes of ram so it's finding a machine to build pipe eyes a little bit of a challenge if you want to do that now what it's actually doing under the covers is it's actually taking pi PI and turning it into C code if you go look at it actually creates 10 and almost ten and a half million lines of C code spread across 800 files it's about 350 megabytes of source that's what you that's actually what it's doing right now is it's basically writing out getting ready to write C code out one one consequence that does take a lot in memory and it might actually break the C compiler I've had that happen on a number of occasions where you compile it totally break to your C compiler in the process of that and you look at that and one of the things that you might look at you would see that and say well that's just insane I'm going to run away from it you can also look at and say well that's just amazing Darius a diabolical you know my favorite word there I mean it's very intimidating part if by pi PI if you're going to do it and so what I wanted to do is actually kind of take you into the bowels of that a little bit and just sort of see what's going on there so so one of the things that that you should know if you're going to tinker with the pi PI project is that it's implemented in a subset of Python called our Python and this is actually something that I found really confusing as an outsider I'd always heard that I'd always heard that ARP that pi PI was implemented in this quote our Python and and I had never really looked at it very deeply and to me I was a lot only our Python is like this special interpreter kind of like sea Python you know it's like a special interpreter or maybe it's instrumented in some way and something like that that's not what's going on at all it turns out that this our Python is just a restricted subset of the Python language that runs as Python code but it's not really but that's about the only similarity and and pinning down what our Python is turns out to be a little bit tricky I'm having a little bit fun of fun at the pi PI guys that they're on their documentation here one of the things that they write is that our Python is basically everything that will go through the translator which is kind of another way of saying that Python is everything that runs without a trace back okay it's not really it's a little bit hard to kind of you know to like pin that down you know I was like okay so you can go reading about this I mean there are start papers and stuff there's you know a fair amount of documentation on what's happening here to be honest I actually find the documentation a little bit hard to read just at first glance because it assumes knowledge of a lot of different parts of the project okay so you might have to read that multiple times there are some papers that kind of give you these high-level pictures but is it but as far as like hacking on pi PI this is only tangentially useful I think you know it's like a high-level view but it's a little bit hard to dig my brain into it if you dare you can go look at the detailed tech report I probably never should have looked at this if you can't read that in the back they can't read it any better in the front but that this is from this is from a tech report from funding I think from the EU so pi PI had some funding to be fair they had to write it like that to get funding so you know like as an X academic if they didn't write it really confusing like that they wouldn't get any money so so so you can look there you can look at the source code but that's also pretty daunting it's like it's like 4,500 Python files 1.25 million lines of source not counting I mean not counting blank lines and it's like just jumping into that is sort of daunting as well there are some blogs about that actually I would pitch these it's sort of a starting point there's a tutorial and writing interpreter with pipe eye that somebody put together there's recent paper sort of talking about that so that so there are some there are some ways to kind of to sort of jump into it and what I thought I would do here instead of just talking about it is just do it I don't know whether anybody downloaded the the source to 2 pie pie this morning but I thought it would just do a demo of like what is this are Python business and what is the deal with our Python the language that makes up this pie pie project so to start that out I'm just going to write a really simple hello program in Python okay so this is okay this is the main program and then if you've done Python programming for a while you'll put the little little check here okay so this is this is very simple little Python program if you if you've coding you should be able to figure that out I'm just going to run it to make sure that it make sure that it works okay Python program works now one of the things that you would ask at this point is how does how does something like that get handled by this pi PI tool and what you know I'm talking about this our Python subset how does that work turns out to use our Python what you need to do is you need to add just a single function to this that looks like this what is going on here is this is sort of just identifying the start of your program it's saying okay where's the entry point to your program and if you've done a little a little bit of programming in like Java or C this is going to start looking very familiar I mean if you know about like C or Java you know that you have to write a main function and you know that the main function has to take like the command-line arguments and you know that it has to return like an integer I mean is this like low-level C code this is what's going on here it's basically we're specifying an entry point there and in order to process this instead of running it I mean this is where a lot of confusion there is no program called our Python like I don't say our Python hello because no such thing exists okay it's like what is that essentially what you have to do is you have to run this through the Translate program that we did earlier so to do that what I'm going to do is just make a I'm going to make a little sim link to that just because it's I don't want to keep typing it all the time so I'm going to I'm just going to take like the Translate script out of the pipe I project and just make it available locally what you can do is you can just say okay let's just translate hello dot PI and if you do that what it's doing at this point is it's taking your Python code and actually literally turn into C code so it's compiling it's actually very intimidating what it's doing here I mean it's like we're only having like a little you know seven line program and it's kind of churning away on it for for a while there you know generating C okay so eighteen seconds later and you can look at this and say okay there's there's mate it basically made a file called hello.c you run it and you get your hello world okay so it essentially it's taking Python code turning into C code to do that it just it's a little as a little bit more sort of complicated version of this okay let's do something very useful this is the most practical application in the world right we're going to well do Fibonacci numbers because it's it's so it's so useful here okay am i I mean is having trouble hitting the ctrl key there okay yet this this is the that the very useful version of this so if we're at a node conference we doing this a sink of course so so the the so what we're going to do here is we'll just print out like the Theia okay there there's like a different version of this so this is this is code which I hope computes like a Fibonacci number on the command line and just to sort of see what this this our Python is giving you I'm going to I'm going to just time like like how long it takes to run this just using like a normal Python the normal Python interpreter okay so this is is going to compute 36 Fibonacci number using a horrible algorithm by the way do that did that on purpose you'll run this and it comes back and says okay takes like eight seconds now if we run this through the through the translate program and wait you know the the obligatory twenty seconds here okay is we got to got to go to watch dad draw start start drawing drawing fractals and things it's a good way to intimidate your friends well okay so 18 seconds what this is going to do is it's going to make a C program that will run the Fibonacci number and we'll just time that 0.1 seven seconds okay so so the big thing the big thing that you're getting looks like a little bit off the side of the screen there is you're getting significantly better performance on that I mean essentially it's taking your Python code and it's taking it to C code in fact you can actually look at that C code if you're if you're extremely insane actually on so if you were to like copy this the directory where it where it put that you will go you can drop down and sort of see a bunch of C files that get they get generated by this I believe that the that the function is actually in this file called implement dot C and you can do a search for the Fibonacci code in there and see what you see what you find that's what it looks like it's not very readable it's got lots of go twos in it wouldn't muster probably code review if you were to write like that but but but it is turning it down into C code underneath the cover now one one of the big picture things about this as well that I wanted to do actually have a C version of this program that I sort of wrote in advance here so that this is just C code that does the same thing if I run this through the C compiler and then time that it's actually slower than what our Python made now okay so now they're not now in the defense of the C compiler I did not specify any optimization okay so if I do some optimization on it I can still come down a bit okay so but but essentially our Python I mean it's it's it's taking Python sitting down to see you're getting performance comparable to C and really the big idea on that project is that they can write the Python interpreter in this Python like language and take it down to something that's comparable to C Python I mean that's really what they're what they're doing with that so so that's a little bit you know a little bit of a taste of of what's going on now there are some things that you that you cannot do I wanted to actually illustrate just a few of the limitations of our Python one of the things that you do not get is you do not get any of the dynamic features of Python there's there's a reason why it's called restricted so for instance if you had a like let's say you have like a list of numbers and 1 2 3 4 and you said 4 X and Nam's you know print X that will actually work fine but if for some reason I were to make that list have something like a string in it that would actually blow up our Python because it essentially it's like a type checked version of Python and it actually expects things to be kind of it consistent type just to sort of see you know what would happen with that just so you know if you have any kind of type error in the system it will actually blow up with a fairly big trace bag that looks something like that I mean it will hit that it takes a little bit of time to kind of to kind of figure out what's going on but it's essentially complaining that you have inconsistent types and the in the type system so so so to kind of sort of summarize this is this sort of aspect of it you know this our Python it's kind of its restricted side of the Python language that they use to implement the Python interpreter there will be a quiz later by the way one other thing that I just wanted to throw in here real quick as well is that this this art python also has the ability to talk to C code because it's kind of going to see there are hooks to make it kind of talk to the rest of the world I thought I'd just sort of show an example of that so you can see what that looks like essentially there is sort of a whole low-level like remote function inner interface if you've ever used something like C types our Python it's actually does something similar to that essentially you can say things hey I want to I want to call like the you know the sine function from the you know the C library and then you can sort of attach like a type signature to it I'm going to keep my fingers crossed that I actually remember this correctly okay so so you can do things you can do things like that I don't know let's just try it okay so what if what I'm doing here is I'm actually declaring like an external function from C and just bring it into my program and then what I'm doing is calculating that down here okay let's let's just try this and see what happens that's another aspect of this of this pie you know the our Python basically the ability to sort of go to the outside world and then call external function it's a built-in it's a key built-in pause for me to rest oh it's not a Python program anymore okay so okay so they are you getting like values of the sign function so that's that's a little bit of a taste of that you know it's essentially they have you know this are Python it's essentially a completely different language almost I mean it's actually it's Python syntax but it's really like working in a compiled language you have to compile it you have to translate it it's a little bit restricted one thing that you should know is that if you love Python you probably hate Arbeit just say that out freud's it's not a very pleasant environment to work in but that that is the language the language of Pi Phi one of the things that is doing just so you know a little illustration of it is it is using something known as type inference to process your code essentially what it does is it looks at your code and it sees how you use tights and it says oh there's an int there oh so that must be an int when you're returning an end and that's an int and you know it's essentially what it's doing is it's looking at your KO and just applying types across it and you really need to think like C programming if you're going to do that and so what ends up happening is you need to think about what happens with this with your whole program essentially what what what our Python is doing is it looks at this whole implementation of pi pi and it just starts walking the code doing type inference so it sort of starts with that main function and you can think about it just exploring like every single branch of your program until it finds all reachable places of code and if you want to imagine sort of you know like one of the things to imagine is like what happens with that if you're doing my tens of thousands of functions that is actually what's going on in that build process is it's sort of walking every single possible control flow branch of the entire pi PI interpreter and figuring out what to do with it it's actually pretty mind-blowing actually that it that it does that just just and sort of a key insight on understanding pi PI is that all of the code that can be reached doing that it's actually what this our Python language pertains to so what ends up happening is it takes the source of Phi Phi gathers the whole thing up translates to C and then writes out an executable so that that's what's going on in the in the build process of Pi by and it's actually I actually find it staggering that there doing that I mean it's so amazing what's what's happening there I think it deserves kind of obligatory like dr. Strangelove reference there if you look at that it's like you know just completely blow your head up sort of looking into that and so so understanding that part of part of pipe I it is it is something that's very very intense fully understanding it is probably very difficult I don't know whether has like souls of PhD students in there or whatever but that that is that's a little bit of a taste of what it's doing one of the other things that I wanted to do is actually just talk a little bit about what it does under the covers I actually got interested in this early on and it was probably a mistake to go digging under that you know lifting up that rock to sort of see what was there there but essentially what one of the things that happens is you might look at that translator and say how does that thing work like like what does it do is it does it take your Python code and does it parse it through a compiler or something like that you mean while you look at that and say oh does it take my functions and run it through like a lexer and a parser and all that turns out that it does not do that what it what it does is it actually it actually knows that it's basically inside it says hey Python already parsed all this code for me why would I do it again and it turns out that it doesn't even operate off source code I mean this is actually very mind-boggling about that that tools it does not it never even looks at your source code what it does is it operates entirely off of code objects now this is a little bit subtle but one of the things that you might know about Python is that when you write functions they get compiled down into byte code and that byte code is what goes in a pyc file amongst other things turns out that that is what that translator tool looks at it looks at byte code and translates it directly from that and so here here's kind of the big picture on it if you know how Python works essentially what Python what C Python does is there's a little bytecode interpreter it runs bytecode okay so it runs those code objects that is what pi PI is doing as well it turns out that in pi PI there is a bytecode interpreter as well it just happens to be written in Python okay that's actually the whole point of the project right is to write Python and Python so there's a Python bytecode interpreter in there and what they do and this is just like utterly mind-boggling it turns out that the pi PI byte code interpreter is modular enough that they literally lift it out and put it into the translate program and then they use it to run your program in the abstract if you think about this too long what you'll end up finding is that pi pi translates itself into see using itself okay so you need like this this is where like a coffee addiction helps you know or 5-hour energy or something like that so it's probably best not to think about it too much but that's that's that that's insane if you look at look at the code and really what it's doing just to kind of give you a sense on it is it's running the bytecode of the program kind of in the abstract instruction by instruction and it just sort of makes it sort of records what ends up happening and then it reconstructs the structure of your program is a huge graph and then it takes that graph and then it drops it into C code I'm going to not go into further details about that other than other than just say that it's really really hairy there okay everybody okay with that and about you in the back you okay with it okay so that's that's that's what's that does that's what it's doing now one of the one of the problems that I'm talking a little bit about some of the problems that I've had trying to figure this out as an outsider coming into PI pi one of the tricky parts is that everything is in Python I don't has anybody ever had that thought like you wake up in the morning you're like boy if you're really cool if everything was by that like the operating system and the compiler and them in the shell and the you know the network stack and the browser and it's like you will love pi pi if that's what you if you ever wake up like that because literally everything is in Python and that actually makes it very difficult to sort your head around it because it turns out that a lot of the implementation you have Python files where there's two different programming languages existing at the same time that both have Python syntax but different semantics okay so you might have some five where it's like full Python and some of it and our Python and other parts of it and you have this question of like well which one is it it's it would be kind of like the same not to pick on PHP it'd be kind of like the same if why PHP had the same syntax as HTML right you know it's like means like how you know like like what well it wasn't meant to be a low blow no this is tricky though I mean it's like if you have like two different languages with the same syntax in the same file that's a little a little bit tricky I can I can tell you what the pipe I guys do these doc strings pretty extensively so you will see ya know I mean it's like they they they will put a doc string in there saying this is not our Python and they will actually it's actually enforced in the translator like if that translate script even hits anything with that in there it will break it'll basically say no you know I'm not going to do it so I mean it's like a I'm gonna feel bad comparing it to PHP well it's okay so that but it but it is you know it's a really tricky thing to wrap your brain around but and then if you even think about it further you sort of think like well why why would you even do that like why would you write code that has like both normal Python and our Python and at the same time I mean it's that that's sort of insanity what's going on here is that there are actually two different execution context for Python for code in pi PI there's basically the context of the translator and there's a context in the final executable and essentially the code separates at translate time into one of those two things and really what's going on here is that the C executable has implementation code and the translator has metaprogramming and what I mean by meta programming is things like decorators metaclasses exact things like that and just as an example I mean this is the kind of thing that you might find I mean this is just you know an example of what a decorator looks like in Python so you maybe you have a decorator function you would actually have this thing split up like that okay we're like the decorator itself is Python but the wrapper is our Python and the application of the decorator is Python and the function is our Python again this is like really we be really tough to wrap your brain around unless you've been like in this project for a long time and I think it's like a really critical thing about understanding the code that's in so so so as far as sorting this out I mean you have to get this sort of mental picture about like okay what happens when like it generally anything that takes place at import time is part of the translator and anything that's that's reachable through that entry function that's what our Python is very difficult to kind of keep it straight the other thing that I would sort of throw out there there's there's even more to this there's a whole aspect of our Python that concerns access to foreign code I gave a little example of that earlier but there's like a configuration system it can do things similar to auto comm not to you know I don't know that you've done auto confidence covery from the C compiler rather than talk about that I thought I'd I should just throw you into some like real art I like real pipe Aiko and just sort see what you found just because this is this is meant to be sort of a diabolical talk why don't we go into the thread library okay so so this is visit this is code from from pi pi so to see what it looks like there's some other challenges too there's a lot of imports a lot of sub modules that you need to to look at but what you will see is things like things like this actually this little bit this little bit of code here this is actually a pi PI source doing auto cough make file stuff it's actually going out and discovering things about the sila about the thread library so you'll see things like it's specifying all with like C header files include directories it's looking for different different things declaring external funds here so here's an example of this like not our Python thing okay so you have some source or the saying okay that is not our Python and this is this is more calling out to see oh the Gil okay let's look at that okay well I'll come back to that okay so but that's you know that this is what a lot of the code you know looks like that so there's a lot of these are some meta programming features if you see a lot of use of decorators basically you know like the in this decorator keep in mind that's part of basically the translator the function is part of our Python again very very difficult thing to kind of kind of sort out so so that's that's that's what a lot of the cravat of the code looks like and how am i doing on time lucky okay so so one of the quandary that I had at this point yes I sort of got to this point in the talk and I'm like how am I going to end this talk because on I sort of feel like I haven't even really told you anything about pi PI like I haven't told you how the JIT works I don't haven't like and I mean is there's a lot of different pieces to it so how long the world am I going to end this thing and then I kind of had this this realization that I just don't even know how it works yeah I mean I feel somewhat I feel sort of somewhat defeated by the thing you know like if you were to ask me well how does the pi PI JIT work no idea I mean other than what they've told me at conferences and so forth so it's a little bit defeated but then I sort of start thinking about in it's like I don't even know how C Python works okay now you could say okay how the hell is this guy giving a keynote talk he doesn't know but it but it's true though it's like like if you were to ask me like about how a lot of different parts of cpython work I don't know like how does like how does a Unicode databases how's that implement I don't know or how's a you know it's like see python is a big program I haven't looked at all that and it's you know it's yet I don't know how all of it works so okay you can say okay how is you know gave up here talk talking about this but I think like the critical thing is that I know how to use the tools that make up see Python that's really the critical aspect of this I know about ANSI C I know about make files and about Auto cough and algorithms and data structures and things and so and so even though I don't know how all of C Python works I know how to go into it and find out how it works so if I want to know how the Gil works I can go in there and find out how it works or if I want to know how you know like you know that like this you know the Reg acts module works or something there's tools to kind of do that and I think that that is really the challenge going into pi PI that you have to kind of overcome it's that that project is basically using a different set of tools than you're used to and unless you're comfortable with those tools it's just going to be this interval fog of stuff and it's in it it's like you really need to know about you know what this are Python thing is you need to know about translation you need to have you need to be really up on Python metaprogramming stuff like decorators and metaclasses and other stuff it's basically a different vocabulary I mean that's the way to think of it and it unless you have that vocabulary you're just not going to be able to mess around with it so so with that said you know this question of how to end the talk I sort of realized that I actually used to be a professor this is actually something that somebody wrote on my whiteboard when I was teaching operating systems not quite sure what to make of that so having been a professor I thought it would end it by just leaving it as an exercise to you to go figure out how to ice so isn't that what professors do when they get near the end of the lecture there's like that or or if they don't know that's even better they leave it as an exercise so so so that's kind of at the end of of that part of the talk I didn't want to put a little bit of a postscript on this whole thing by talking about Ruby like hey we're late where is this going okay so yeah Ruby one of the things that you know is that I like to break global interpreter lock Sal OTT you know I wake up in the morning it's the first thing I think about Lois no not really but I've done a lot of work like looking at the Python Gil and and things like that and one of the things that kind of noticed I did it there's a benchmarking on some thread stuff I came across this benchmark where it's like okay Ruby 3,600 times slower than Python okay on some thread benchmark and I'm not willing okay that I know I'm not willing to accept some weird result about Ruby and just say okay Ruby socks or something actually so if something is going on with that and the question is what is it okay and I did that I mean actually I was invited to speak at the root PI conference which is like a Ruby Python mixed conference I thought it would be kind of cool let's go look at the Ruby Gill I'm not a ruby expert I downloaded the source code for it but I was able to kind of look at their their their Gil in pretty short order like an hour or two it's like find ins like no okay and that's around with it I will say that that performance problem they have there is a more extreme example of a performance problem that Python 3 3 has that's a topic for a different talk but it's a you know I went in there and filled it around with it and just to be clear about that I could not write a ruby program right now to save my life okay I mean if you ask me Oh Dave you're gonna die unless you write like some ruby program up here that'd be over to game over so but but but the thing is I had like basically the vocabulary to go in there and understand it like basically C code and how to tinker with it how to look at it and the whole reason that I did this talk is that I encountered that same problem in pi PI it was basically totally lost with trying to figure out why I mean it was you know I had a benchmark with pi PI maybe a year ago similar performance I went to look at it and just felt completely beat up about it I mean I went into the source it's like I'm Way out of my league here because I just don't have any basis for understanding what's happening and so that was really kind of you know that the the maybe the inception of this talk is you know I'm going to make maybe I need to you know do a little side project to figure out what is going on with PI by to see whether you can tinker with it and you know and by presenting it may be you know sort of having other people kind of get a little bit of a you know a head start or a jump on it if you wanted to do it yourself so so some partying where it's on the talk you know this question of can you tinker with pi PI I'm not sure okay I'll I'll be honest like you know it's an it's a non-trivial thing to go in there but I think you should do it anyway partly because it's a cool project you'll you'll see a lot of neat things in there they need something need more love to you know any more more people on IRC asking questions and stuff so there's a lot of neat things in there if you go in there you might find some things that they're kind of wild or vm's I don't know as full of stars or BM so so that's kind of the end of the talk I guess my hope is that you learn at least one new thing by sort of going through that or at least just kind of an idea of what that project involves if you ever want to mess around with it just some just some special thanks Alex Gaynor and magic did help me out with some occasional questions on Twitter and things like that I also tried this out at the chippy meeting for a couple of couple of months there's some there's some actually some extra talks you can find online that has been going into more detail behind the scenes of what's going on if you're interested in that you can look at that so I think that's that's it I don't know where I am on time but we can probably we have time for questions or no time for what it's going to ask me about copywriter all right thank you very much with a pepper sauce okay

Channel: Next Day Video

Views: 31,725

Keywords: psf, python pycon pycon2012, pycon_2012

Id: l_HBRhcgeuQ

Length: 43min 23sec (2603 seconds)

Published: Sun Mar 11 2012