Raymond Hettinger, "Being a Core Developer in Python", PyBay2016

hello how are you guys doing all right so had to think of a keynote topic and thought what is something that I uniquely know and the answer is I've been a core developer for 16 years or just coming up on 16 years now and so I could give you an insight into that world so I've organized this into a handful of parts one part is I just want to tell you some anecdotes and stories from core developer world so much I'm hoping it'll be a little bit amusing then I'll tell you some of the facts of life and in that world and invite you into it and give you some advice for coming into it and how to participate and how many of you know what I do for a living right now I teach Python and you know the problem with being a teacher is you can't turn it off which means that there's lessons in here and then after the lessons there's a test I'm not kidding the back half of the presentation is a case study a series of case studies and the answers to the case studies are being given in advance so all the parts that seem like little breezy points will see if you've all learned them all there's a test at the end fair enough all right I'm Raymond heading toward this is my Twitter account Raymond H I don't tweet when I get off of an airplane or see a cool movie I tweet about Python I teach Python through Twitter and so I hope you guys follow if only to learn a little bit more about other language thank you for inviting me to this conference thank you for the sponsors who put this together Microsoft was amazing and providing a space yesterday for the tutorial we're very very lucky to have had these facilities so anecdotes some stories about my life and open-source just some amusing times it all started with Oz Khan at least the speaking part of my role about seven years ago I spoke to an audience that had maybe about 15 people in it Alex was in that audience so was a quitter van rossum so were several friends that I knew I knew half of the people in the audience do you think that helped it did not I shook the whole time mortal fear a few months later I went to Europe icon and I gave a five minute lightning talk no pressure rights only five minutes I shook the whole time and almost froze just a mortal fear of public speaking that was a few years ago I've gotten over it okay once upon a time I boarded a train in Shanghai if one of the nice things about the Python world is it's very large I got invited to teach a class over there and one of the things I like to do when I travel is get on the public transportation go around explore myself and I'd not see the tourist side of the world but just see the rest of the world so it's a simple thing and everywhere you go you get on the public transportation and you learn something about how the people live there and whatnot and so it's a simple thing I walk out there's a train pointing into downtown Shanghai it has a bunch of Chinese characters on it which surely meant going to downtown Shanghai I got on the train the door closed I have things normal so far and then the train did something interesting it made a turn and left Shanghai now keep in mind I couldn't read anything that was on there now Shanghai is getting further and further away many places in the world I go English will get you by and sign language not Shanghai Wow I was kind of in trouble there was a chance that I was going to never be seen or heard from ever again I started looking around looking for resources and then I see it at the end of the Train there's a young man who has a book in his hand Alex martelli's Python cookbook I walked over to this person opened up the book it's around chapter 18 and Alex invited me to write one or two chapters in that book opened it up pointed my name and said that's me I made a new friend there's two Chinese symbols I used to be able to write he taught me four it translates roughly to famous person okay my new best friend took me into a downtown Shanghai showed me all around and guess who got some free Python lessons and advice and code reviews and whatnot and thanks for saving me from being lost in China so a Python world is actually a very very large and interesting and thank you Alex you saved me from being lost in China all right another experience I went to pike hunt there's lots of them now and they're fairly big this one might not seem huge to you but this was about the size of Python conferences not that many years ago and I can't remember which exactly which one it might have been the first one in Dallas which was not much bigger than this and I went to one of the talks and there was a fellow named Jim Baker and he was giving a Python on talk on the itertools module and all kinds of ways to use it and how awesome it was and that was a wonderful experience for me I had never gone to a talk listening to somebody talk about something that I had made before and it's a heady experience now I don't know if there's any lesson in that it's just the first time you sit there and hear somebody give a talk authoritative talk on something you wrote you wow this is cool and then every now and then you learn something anyways like I didn't know that which is astonishing Peter Norvig a personal hero I have this thought of if you would like to improve your intelligence and prove your smarts all you have to do is say Peter Norvig zni and your IQ points go up a few point seriously I believe if you read basically anything that he's ever published it improves your skills quite a bit and he is a fair number of works on Python and so I've one of his more recent ones on all probability it took a look at and substantially everything in it used tools that I wrote which was a big deal for me obviously Wow Peter uses my stuff to express ideas this is all really cool isn't it great when somebody uses your stuff to do something interesting yeah oh you might think before I became a core developer I published a module oh I wanted to use something like numpy but I was a Windows programmer at the time didn't have access to a C compiler and Fortran compiler I needed to build it there was not good distribution tools at the time and so if you need a matrix package and you don't have one what do you do you're a computer programmer what do you do you write a computer program so I broke Matt funk py and this was my first big package and I tried to be very careful I put footnotes all all throughout for every algorithm where I got it from which textbook it came from I tried to research the best approaches each problem make very clean code and test it put in preconditions and postconditions and whatnot wrote up a really nice document fancied myself as kind of a you know pseudo academic with a little paper around it and I put it on my personal website and the interesting thing about websites at the time was analytics were pretty rare at that point if you put something on the website you have no idea if anybody is even looking at stuff you put up the experience at the time of having a personal website was stepping into a dark room and talking and telling jokes describing things that you know and there's complete silence in the room unless one day you say something to this dark room and all of a sudden you hear a lot of applause and that's the only way that you would know that you had any users so I put up this package and I got no feedback at all just say it for two years then I got one email it was from CERN and it was a feature request and then along with this feature request they just put in their everybody at CERN is using Matt funk py wheel of Python LC for God's sake still use this for anything nuclear is my first baguette I met the outer limits of my mathematical knowledge putting this thing together anyway that one scared the bejesus out of me went to another Python conference at one point and there's lots of swag and people are giving out t-shirts and whatnot and every now and then I will see a teacher and add some code on it and the code has something I wrote which feels kind of cool so I now walk up to people and say I wrote that code and then they give me a t-shirt so I've got a nice collection of which raises a question what do you do with all of these t-shirts who knows the best answer make a quilt is in fact the right answer and that was lit was it Lin root oh you don't know this has been done Wow so I thought you were so I think I think it was loud Lin root just collected together all of her t-shirts from all the conference's you've been to and she made a quilt out of it so you independently came up with that brother than yes it's the right answer all right and then cool things being invited alla Brownlow our world I gave a talk in a keynote in Pune India which was a really cool I went out there was a thousand people in the audience I had never seen any of these folks before I pound on the table what did they say the meme made it all the way to India which is a fantastic thing and there's like would you like to give a keynote at Moscow I'm like sure I'll go to Moscow and stay there for a while and take my family with me and we'll travel all through Russia and so that was an adventure afforded to me through the Python world uh other aspects of the Python world at one time we had a mailing list a Python a tutor list and that's still available and the Python tutor list is a way where people help each other out and it's famous for being very kind and friendly there's a disadvantaged tour of it though and that all of your posts are public what if you need some private and one-on-one help and coaching we had a separate mail list for that it was Python help and so python help someone could send it to an individual a note to an individual core developer and get individual help and so we took turns meaning the help desk and got notes from people all around the world and would help them with their problems now different things can come out of this depending on what your shift is for the help desk and a matter of luck so let's see what kind of things can come to you out of being on the help desk you could be let's say Alex martelly you're manning the help desk and a Ravenscroft sent you an email you send her an email back she sent you an email back you sent her an email back next thing you know you're at your wedding the reading as in a Python that's what happens when Alex mans the help desk mine the very first one I got was my Python has six eggs at what temperature do I keep the eggs in order at which one I have to refer them to the herpetology a new script Alex gets a wife I get Python eggs just saying so that's how it rolls and those are adventures from a Python land I just thought I'd share some of the little ups and downs and fun parts and they tree does it sound cool although the delicious perks of being a Python core developer yeah good news we're hiring now I don't know how to write employment ads so I just took a look at the ads that are currently being posted and I thought you know hey I'll use these as a model so I'll tell you a little bit about what we're looking for diverse developers wanted if you're diverse we want you if you're not diverse we want you will take anybody the attraction to the company come build software that changes the world that was not the original traction because when I joined Python it wasn't that popular not that popular at all fact I first heard about Python when I was taking a MATLAB class in Natick Massachusetts and one of my classmates um it's mentioned it to me you can work from home on a distributed collaborative team pretty nice uh and this is the best part lots of companies have rules but we don't you can work as many hours as you want and nobody will make you stop now some companies you join the company you want to work on exciting project X but they need to get something else off the door and they shift you to the project that's no fun for us you can choose any of the projects on Python that you want to work on if you oh I want to work on the elementtree package why you can just go right ahead isn't that great free to choose you work from homes many's hours you want and the best part is the pay I have been there sixteen years and my salary has doubled every single year and I can promise the same for you well there was one time where I stepped on quito's toes and he cut my salary in half that year but I eventually got over it and I'm back up to where I was before just saying how's that for a job ad all right it's funny but it's actually all true all of it which raises the question what can you do if you want to participate in a Python core development do I even recommend that you become a Python core developer no it is not the greatest job it is thankless it pays very little it can consume an ordinate amounts of your time and a lot of times it's not particularly fun and if you want to change the world if you want to get your code out there and whatnot the best thing you can do is join a project that uses Python those projects need you python core is fairly mature of source and so you're going to find the in a mature codebase almost everything you touch and must every idea you have might make it worse in some way and yet if you get your experience on another project one that needs you almost every day idea you have will improve the project and you can be a lot more productive at the outset and then move to something like the Python core so I don't think it's a particularly good place to start fairly complicated code base its enormous do you think have you studied at 16 years you might know at all how about 26 years who studied the Python code base for 26 years Guido van Rossum the person wrote it he uses Python every day he oversees it's a development he speaks about he makes his living doing Python do you think he knows it pretty well it is my belief that he knows less than 1% of what's going on in the Python world it is my belief that he has seen less than 25% of the code in the core it is bigger than any one human being can fully comprehend the rate of if you looked at my emails it'd be insane you would see on the order of fifteen thousand emails a month adar uh Python related I can I track every Python check-in I read every post to the bug tracker I used to read Python dev all the time until people started talking too much and that's a bit of a joke at the same time some chatty and happy people join Python dev it rendered the channel unusable the number of emails went up by about a factor of five in a short period of time at which point you just can't devote two hours a day to reading just one mail list you actually have to go out and do some work and it's better to read the bug tracker and so there's an cuido himself does not follow all of the posts on the bug tracker he does follow Python dev but he can't even follow all the check-ins it's bigger than any one human being can comprehend in fact it is so we need your help here's some things that you can do oh if you wanted to help the elementtree package doesn't have doc strings in it is that a pretty easy thing to do sure what do you need to do go get good at elementtree work with it a little bit look at some published code read the documentation think about it come to understand it deeply and then try and translate the documentation into good doc strings with some effort that is something that pretty much anyone in this room can do and it would instantly make Python better and more usable particularly for people who use iron ipython they really rely on this help rather than the published documentation another thing you can do suppose you have a lot of free time you're an amazingly good seed programmer and you don't have any social life whatsoever and you're trying to think how can I eat up all of my time or all of you in that situation okay suppose you had about a week week and a half what you could do is there's a patch out there right now on this issue or a compact dictionary it will make Python better the dictionaries will be smaller than our current dictionaries they will iterate much faster the patch is ready it has tests it has timings it's a clean patch it's a good patch by somebody who put a lot of effort into this and it's modeled after something successful the compact dick that is in pi pi that was based on a proof of concept code that i published on somewhere at one point packed they were initially calling it the raymond dict the name did not a stick come back dick is a better name for it so is it ready to go oh the code needs to be reviewed somebody has to do it somebody had to you see and I picked somebody I don't know how to make this part rhyme we read a lot of dr. Seuss at my house okay and so how did it go Rachel she doesn't remember that so somebody had to somebody had to you see so my mama picked somebody Sally and me okay and I'm picking you and Sally to go review the compact dick it is not an easy thing it changes a lot of code each of the lines might have a profound implication for performance or security in Python it is an easy thing to get wrong it really needs to be reviewed even though it's good if you were to review it we can put it in possibly we could put it into Python three six but right now not any of us has the spare clock cycle still put in the amount of effort that it is going to take that's something you could do right away when people become Python core developers often they want to put new stuff in Python really what they should be doing is taking the existing stuff and making it better the place where I believe the language is most efficient is in the quality of error messages I notice this a lot because I teach Python and people get error messages that are nonsensical to them it doesn't say you made this mistake here's what I really wanted here's what you probably meant go do this it'll fix it those are really helpful error messages unhelpful error messages is I expected a buffer object I refused to do the action that you are requesting we have lots of error messages like that getting those fixed is not a matter of just changing the text a lot of them you have to be able to analyze the context that the error happened in and there understand where the users can go wrong one of the things that the experienced Python core developers have a hard time with is we are so used to using the language correctly it becomes fairly difficult over time to conceive a people's profoundly different mental models of what they would do to wreck it and get it into a bad situation so it's almost hard to conceive at the time of your writing something what is a really good error message to put in that's it a lot of you are in a position to be able to come up with those things if you find something that is hazardous to someone's health error message wise submit a patch it's very welcome idle has the letters ID and e in it because it's an IDE it's actually named for Eric Idle from Monty Python's of a flying circus it is not the world's greatest editor it crashes it has performance problems and yet a lot of people use it because it ships with Python here's what we can do a teach people Python on idle and then till go get yourself a real editor our B we could turn idle into a real editor most of the people in this room are capable of helping do that project I say make it awesome another thing you can do is just weigh in on whether proposals are good or bad a lot of people think that contributing the Python core has a lot to do with writing patches I don't think so I think the patches are the least important part we'll talk about some of those in the case study an example of current moment is in the random module there's a function called choice and what it does is take out of a population and chooses an element randomly I have a population of people in this room random choice choose one of them the proposal is let's make a weighted choice so that the people in this half of the room have twice the chance of getting selected as people on this part of other room fair enough are in The Hunger Games the older you get the more lottery tickets you have inside to get chosen for other games a lottery you don't want to win unless it's to save your little sister and you get a chance to shoot at people with a bow and arrow okay all right that's somewhere else so you can weigh in on proposals like that howdy could you participate in that could you write me a patch for it no I don't want a patch why not the answer is I believe most people in this room are capable of writing that function and quickly most of you could knock something out in five minutes in 15 minutes you can make something fairly good-looking in an hour you can make something good-looking good variable names nice doc string doc patch and test cases any of us can do it and because any of us can do it quickly it suggests that there's that's not where the worth is I'd like you to do the hard part I'd like you to help me with the hard part the hard part is not making that patch any of us could do that that's effortless what is the hard part the hard part is what is the right API for it should it return a generator well that would be kind of cool it would run really fast but nothing else in the random module looks like that so it's non harmonious with the rest of the module how many of you use the random module all the time how many of you do data science okay how many of you know how to do Bayesian Monte Carlo simulations using the random module all right you folks matter you matter a lot because a lot of folks who tried to participate in this have essentially no mathematical background at all in other words they are quite far divorced from the people who actually use this so it's pretty easy for them to design a crap API I know we'll pass in a dictionary that has the population as the keys and the weights as a value well this is great do you think somebody out there might have some population members that aren't hashable that's I a problem do you think data arrives in the form of a dictionary maybe it does maybe it doesn't if you can't enter an exact quick toy example might be effortless it might be that the norm is that the population and the weights arrived separately it might be that the norm is you rerun with the same population many different times but with different weights I don't know what that norm is nor do most of the other people who participated in this discussion you guys know the answers to those questions go out and look at actual use cases talk to people who are applied mathematicians statisticians and data scientists get their input for it take some use cases take ten different variants of the API and try those use cases with them and see which one fits the best that's useful that's something any of you can do and you don't have to know a lot of Python to do it and that will make Python better one of the challenges of the standard library is we put something in it becomes standard by definition which means it's really hard to change a lot of people rely on it and so if you put in something with a bad API we have to live with it for all eternity you wouldn't want that would you that would be a terrible thing all right so this is a call to action people are often saying what can i what can I do here are some specific things you can do concluding on issue numbers so that you could go chime in right now Jessica last night made a nice call to action she said learn the rules of the game use those rules to go change the game and improve the world I'm showing you the rules of our game and I'm giving you an on-ramp point so that you can join and go help change the world and next time you're stuck in China you can go to a book and point inside at your name and say I wrote that fair enough are better yet I design that all right so let's go work on your management skills oh I was reviewing a book by Kenneth Wright's it's actually a community book but he's the lead author on it Hitchhiker's Guide to Python and this jumped out right at the beginning I thought I would share it with you it's a variation of the spider-man rule that says with great power comes great responsibility a real pythonista knows how to write metaclasses and decorators and all kinds of python foo and once in a while we use it to take a deadly problem and we smack it down using these advanced tools the other 99.99% of the time if you're a real Python ista you don't whip out all of these tools you use for loops use a simple class you code in the normal way and don't do anything exotic in one of our problems is has been some of our more core developers got to the point were they knew new kung-fu then they joined the Python core development team and they started using it not on important problems but on small problems so I'm going to pick on one out that is this is me just being a little opinionated but it'll share with you a thought of ways I think that our development process can go wrong so a person joined the Python core and it's not just one person but I was just say there's a group of people involved here and all of them knew how to do the kung-fu now we have a small problem a very small problem the very small problem is this if you define a a constant in Python red is equal to 1 and later you print out red it shows as the number 1 not as the word red how catastrophic of a problem is this well Python got by 24 years without addressing this problem once in a while people would set red equal to the string red and it would work just fine there were many solutions to the problem but in other languages we have enum so you see other languages have enum you come to Python you see this problem it's logical to put those two concepts together and say lots of other languages have one we have this problem it's a natural real problem we should put the two together now you whip out your kung-fu and you put together an enum class the inyoung class remember the problem it solves is it makes the word red print out is color dot red instead of as one that's the only problem it solves involve if you were to go look at the source code for e num it's got metaclasses in it it's got descriptors it's got hooks on set attribute it has pretty much every trick in the book that I teach in advance Python classes and that's an insane amount of firepower at a small problem languages that have an enum mostly only have one they just say here's how you do an e note we've got two variants but inside the documentation there's another 10 variants and another one almost made it in this way I shouldn't say almost made it in it got a god in this week I'll show it to you in our case study section in a few minutes and I had some success in last night the developer really stepped up to the plate it's several if you think it's an act of courage to check in code it's a bigger act of courage to acknowledge maybe this wasn't such a good idea and pull it out just before a release and that happened just last night so I applaud the developer for their courage and saying whoops that was a bridge too far fair enough all right so uh I think this is fairly important in our world oh it is quite different when you're developing tools that use Python because you can make mistakes and undo them later but if you put them in the Python core it tends to be hard to undo fair enough anyway that's a can can its lot since you guys know who guinness reaches author of the request module by ton superstar okay fair enough all right some advice on if you start to participate in our world be a good neighbor this is actually harder than it sounds because you step in to the Python world and you want to start to participate any like there's a flower garden I should go step in that flower garden and rearrange all the flowers and trample a few of them there I'm contributing some of those flower gardens are carefully maintained by people who only participate because they like other flower garden looks and it works perfectly for a set of people and if you were to go rearrange the flowers even if you make an improvement to the flower bed possibly you've completely undermined somebody's motivation for participating an example would be the JSON module the core developer who submitted that who originally wrote simple JSON and then donated it to a Python had a very clear worldview of what it was supposed to do had a very large user base that was very happy with it it's not particularly active suppose you joined the Python core and go in and say oh I think the JSON module should do something differently now keep in mind this person has a job very little spare time and you go in and rearrange all their code even if you improve the JSON a little bit what do you think it you did about that the effect that you had on that person's willingness to maintain that module in the future so in fact that is the thing that has happened is periodically people come in and in just trying to help they stamp step on other people's flower gardens fair enough all right how another related concept here of trying to think of some way to get rid of this terrible entirely inappropriate joke but I will just go for it it was one that I heard as a kid how many of you've ever heard Helen Keller jokes okay so you already know it's going to be a tasteless what did Helen Keller's parents do when they were mad at her the answer is they would rearrange the furniture in the house isn't that terrible so in fact that would be a mean and cruel thing to do you would never do that would you would you ever refactor code that's not yours you're doing exactly that we've all been taught that refactoring is good and in fact it is mostly a good thing but I want to suggest a alternative point of view that I've arrived at after a lot of thought a lot of years years of of work and I finally think I figured out the core issue with some types of refactoring I took a read a book on improving your memory and a technique that is very old is called the method of loci the Wow and so the Greek and Roman orators would take the parts of their speeches and memorize them by associating each part of the speech with a section of a house or a walk through a city and then to play it back in their mind they mentally redo the walk it occurred to me later the significance of this is our minds have an enormous chunk of our brain is set aside for spatial reasoning I thought about putting an example up here of taking some code you were unfamiliar with and laying it all out on the screen and boxes and saying now this piece on the Left calls this one on the right the one on the bottom loops back in and then this one on the bottom right Zing's in and out several times it is just a support function once in a while we swap all of this up and this piece becomes dominant and in my intent was to take that code explain it to you but use words very much like the words I just use then I would ask does everybody in the room understand the code and my goal was to get you to understand that code then I'm going to show you an improvement to the code where somebody comes in and refactor zit and that refactoring changed the structure the organization of the code and moves it around I'm asking you to come back to the code that you've invested time to understand and say can you manipulate it do you understand it now without reinvesting new time and the answer is often no and so I worked at a company once where we had a refactor err not and refactoring is general good but his thought was as I go to understand code that I haven't seen before what I'll do is just constantly rearrange it in a way that makes sense to me oh well these are related thoughts I'll move them together and the problem is he was very effective for this person it was great for them because at the end they completely understood the code they touched it they moved around organized it to their way of thinking but in their wake not a single person who had ever worked on that code knew where anything was anymore and when you rearrange it it becomes that Helen Keller joke I mentioned that because we have a core developer right now who has an insane amount of time on their hands and insane and is spending a lot of time it's they don't have any external projects that they maintain or whatnot I don't know that they've ever met a user ever person is in a very very good programmer C programmer and their refactoring things in Python lots of them I think if we give this person another year half of the code base will have been touched and rearranged oh I saw this thing mentioned a couple times we should put a macro in and in Vince a brand new macro you know this thing was used twice in one place will pull this code out each one of those things individually is fine collectively the entire code base is becoming unrecognizable to the people who made that code who have less time to contribute as we're going on so when I say don't trample in other people's flower gardens I'm saying that it's possible come in being very enthusiastic make actually things that are individually improvements and to actually destroy the whole community you wouldn't want to do that would you that'd be you're not a flower garden ler no and you wouldn't rearrange Helen Keller as our furniture fair enough so you're not going to go in and take all my cut and refactor it just because it looks a little nicer to you which she would as I know how she is all right picture straighteners you walk by pictures crooked what do you do straight now we all have a little OCD in us as a programmers because if you don't your code ends up looking like a garbage pail interview in the networking world yeah so in the networking world is very interesting you walk by somebody's data center and you see this giant rat's nest of Ethernet cable going all over the place anybody ever seen something like that how confident are you in that person's setup it's indicative of how those people think how they organize how they clean things up how they plan and how they maintain when you see that rat's nest it's a warning sign so when you go into a data center of all the cables are nice and neat and they all go to specific spots and they have nice color codes and it it gives you a great deal more confidence fair enough so if you're a decent programmer you tend to be a picture a picture straightener because you like things to be a little bit neater because they're easier to reason about fair enough on the other hand what if you go around straightening other people's pictures all the time without looking at the photo that's actually quite common in the Python world lots of people come look at the Python cords oh I'm going to just change this code this would be a little bit better this way and I'd like to give you an example of me being victimized by a picture a picture straightener okay and I'm not picking on people here I'm suggesting ways to be effective in big projects and mature projects everything I'm saying here would apply to Linux as well you go join the Linux kernel and you go rearrange all the code you will destroy that project if you go straight in all the pictures like this you can possibly muck up Linux in a fairly big way so I actually have this on a separate slide all right so I had a test in test sets and what it said was assert true that set a is less than or equal to set B easy enough and the assertion passed what happens a picture straightener came by and said long after I wrote this test some new specific asserts were added to the unit test module so everywhere you said a is less than or equal to B there is now an assertion for it and says assert less than okay and so the motivation for this is if we you assert true a is less than B and you get an error message all it says is this is false but if you have an assert self a certain less than what it can do is actually show you the two values and give you a more helpful error message when it falls through so does it make sense that we sometimes change a less-than test to a self assert a less than that's the motivation but the person didn't look at my picture or understand what I was trying to do at the time what did they come through and do there uh that test was not testing less than and sets less than equal means a subset test and so these words are now the wrong words the error message is now the wrong error message and it came up in a bit of a whole bunch of tests in a section where I was testing operators I didn't even want to test subset eNOS I wanted to make sure the less than equal operator actually works so this test now no longer does what I expected it to do now keep in mind that this is the person who I mentioned earlier it has a lot of time on their hands they didn't do this once they've easily changed on the order of five six thousand tests in the standard library going through and making them into specific asserts they believe that they have made it better but in fact far worse and so I don't want to pinnacle in that particular case as much as just say when you come to a big project don't be a picture straightener that doesn't help don't be a weirdo Python attracts weirdos lots of them and I'm not going to say that I'm not one of them and it's actually okay to be a little bit weird we encourage that even by the name of the language but I'd like you to self-censor your ideas if your ideas are just completely random whatever jump popped into your mind it shouldn't immediately show up on the bug tracker as a feature request do a little self-censorship try out your own idea and see if it actually makes an improvement run it past another person and so we have some people out there who don't self censor and they put in many many many feature requests and my problem is I can I can't just shut them down because possibly one in a hundred of them is going to be a good idea but it means that I am now in the role of censoring that person's thoughts it's whereas most of you would actually say is this the idea actually good before you go suggest it you remember Jessica McKeller saw her first contribution to twisted the one that she described yesterday she said that she came she put so much thought into making sure it was perfect that it complied with a the guy that was actually useful fewer than one proposer in a thousand that comes to the Python world did but Jessica does so be Jessica don't be a weirdo don't talk too much on the mail list that I band with is a fine it and don't rush to make a patch it's the least important part Oh finding your way positive advice work on big problems problems people actually I have one of our challenges as compared to other open source projects is the work on some specific tool it's actually fairly well known in advance what problem at school was having but a general-purpose programming language can do anything so we don't actually know in advance what problem we're solving because of that oh it's very easy to invent problems we don't have and create what my wife calls swirling around and so find big problems where people are actually using Python have a real problem and the languages in supporting those tend to be very useful contributions this is very important assume the people who came before you knew what they were doing I came in the footsteps of people like Alex and Tim Peters and Fred Friedrich lund mental Giants and I had a great deal of respect for their code and I learned for them we have a new generation of core devs who presume that everyone who came before them was stupid and must have been wrong and they just start changing their code and if we're not there to take a look it changes fairly rapidly it's been a long time thinking that is mostly how you can contribute is think about a world and this is the hard part you are an atypical user if you are a core developer in fact other users have a profoundly different worldview and you're not making Python for yourself you're making it for other folks and lastly what is the world of being a Python core developer it is a long journey to discover the Zen and harmony of the language it is a beautiful thing it is successful for a reason and as those of you went to my tutorial learned you know that that Z Oz in goes very very deeply and that people after years are still discovering the internal harmony knowing that we should recognize that harmony is a very fragile thing take any song and randomly change a few notes and you destroy the song Python is quite fragile and your random ideas could actually make it worse rather than a better so treat all this is a journey work on big problems assume the people who came before you were smart and think a lot and you can contribute a lot to our Python world thank you very much for inviting me do we have room for a few questions how many how many minutes oh it was a trick question I did actually trick him on purpose I had some case studies I wanted to show you remember I promised you there would be a test let's take a look at API is in Python in the string object you can look for a substring and you can tell it where to start and we're just in the search there's a legitimate use case for that we do repeated index searches into code find also has that what is missing from starts with and ends with let's see if you can pattern match what is are not alike in these patterns start and end who one of these things is not like the other what should we do hyper generalization was the hint to you don't do this we had no actual use case for it no one had ever requested it but instead a new core dev comes along sees four of these have starting in and two of them dumb so they move this out down here what is the consequence to you at the time there was no consequence but later we got an actual use case for something important and that was that what if you have multiple prefixes or suffixes you want to stand scan for dot HTML CSS now we've got a problem these other two positional arguments have been taken up and so now if you want multiple prefixes you have to pass it in as a tuple which always looks weird and it's always difficult to teach this person put extra code in that we have to maintain that is not used by anyone and block actual useful things that we wanted to do you're not going to be a hyper generalizer are you all right enum this is beautiful how do you make a color you could say I want 3 enumerations red green and blue it looks like enum in almost every other language I've seen do you like it it's a beautiful thing there is no doubt about that and it solves a problem that before this you actually had to say red equal one green equal to and blue equal three even if you weren't going to use those numbers we have a problem it's a small problem a person has to do a little typing typing in numbers this went in is there any reason to undo it this is a test to see if you're going to be any good at this these are the decisions you face every day as a core dev and they're easy to get wrong there's something critically wrong with this it took me a while to spot it you guys have one minute what's that okay one thing that you notice is the property is a built in so it turns out that we have to pass in and ignore to it until it to ignore actual Python words and the class definition that's a hint that you're going across the grain of the language that you're fighting the language and you're having to push the rest of the language out of the way if you have to push the rest of the language out of the way it's a hint that you're doing it wrong so in fact that's a hint but notwithstanding that problem there's a problem with this as well everywhere else in the Python world if you have code like this in a class it's an error it's an error because you have a variable without an assignment to it and what that variable will normally does is it's an expression an expression says look up that variable in this case that writing the variable name even when it doesn't already exist creates the variable and does an assignment that's an assigning statement and it assigns one to read automatically nothing else in the language works that like that it breaks not only our mental models it breaks all of our lint tools that would flag this as AI error fortunately the dev who put this in saw the light and reverted it last night if you would like to join our world you need to be the kind of person who doesn't just say oh that's cool let's put it in and actually thinks about this at some depth that is a impact on the rest of the world and it's a harmony with the rest of the universe and I think if you can train yourself to see what's wrong with that and to argue against it and to preserve the purr harmony of haikon I would love to have you as a core dev fair enough all right questions suggestions comments or complaints just the this man she was asking a question I didn't really understand she said I want to know about the code review and she said somewhere it sounded vaguely like the word process I have no idea what that word means we are not a company we're a group of volunteers we are a ragtag group of people some of whom who have no job some people learned a little Python and wanted to contribute to the core we our problem is the more process that you layer on the more you disincentivize people to volunteer their time to go through we want to have code review but how do you review code that you don't understand or it's not in your area of interest someone comes in and makes some sophisticated adjustments to xml module i don't know how to review that without studying the XML spec so we do have a code review but I would hesitate to call it a process it is a very haphazard thing if you're interested in something a patch goes up and you have something you can say about it useful in general we do it's hard to have a more disciplined process than that because people have very very different talent sets different time availabilities different levels of interest and even Cueto himself is only interested in tiny subsets of the language excellent question five bucks thank you guys money one more yes zoom you're obviously very still active in core development what are some of the problems that you are facing right now with core development with the core development process are with the with our large problems that we're actually working on in terms of implementing new features and whatnot yes second one oh so I tried to put some of the problems up here that we could work on how one of them that I'm working on right now is the design of a rated weighted random choice I've sent emails to several applied mathematicians and data scientists trying to get their suggestions in for the design I've asked all of you to participate and that's how I'm approaching that design problem a problem we have right is the compact dick there's a patch ready to go and no one has a week and a half of their life to devote to making sure it's correct before it goes in these are the real problems that we have I like to call them the Star Trek problems if you ever are fans of the old Star Trek what was the theme of every other episode not enough time not enough power it'll take ten days to make it Starbase four we have to make it in two or someone dies I need one hundred and ninety percent power right now the engines cannot take no more we are bandwidth limited in the amount of time that we have to contribute to Python all of us would like to have more hours in the day to contribute we don't have enough hence the call we're hiring okay and one last question and then we'll call it a done deal and we will have the room completely vacated by 11:00 yeah we're starting to vacate in about five minutes hey Raymond I'm feeling free to trample on the accomplice organizers a little bit because they gave me this big block of time and then took 12 minutes right at the beginning of the talk talking about logistics yeah mm-hmm any more questions not from you Cal hi Raymond great talk and how do you believe we could get more of these critical thinking ideas into the core development world but from individuals that aren't currently core developers but might have a lot of product I mean a real-world experience in certain areas this is a fantastic question let me rephrase the question another way to si questions Raymond what was this entire talk all about I'm inviting you into our world I want you to help on these problems I just put a list of them up that all of you could start working on right now if you wanted to you don't have to know the cpython core in order to do it the compact dict yes but several the other things I listed you could work on right now when I said we're hiring it was funny but it's true come join us contribute now excellent question 25 bucks I do okay yes Anna just so you know my first contribution was actually saying I'm sorry but the FAQ that tells people how to figure out why their float division is going wrong has the wrong question on it because anyone who understands what a float is won't need to ask this question so you don't have to have the answer to contribute you just have to have the question about the part of Python that you are using right now and say this is a pain point please help and that gives the real-world use case so they know how to fix it 15 bucks the little Star Trek the needs of the many outweigh the needs of the few that is not what Spock said he said the opposite the needs of the many the users outweigh the needs of the standard library itself that the people who are core devs actually have far different needs than most normal people we don't think about the problems the same way we don't tend to even have the same problems that normal folks do which means we are in dire need of normal people if you are a normal person you can do what Anna did and say a normal person would not even know how to phrase that question here's the problem they would have that would lead you to the FAQ entry so in fact that is the case since I have a couple more minutes things uh when you come come to Python and want to help stay true to a modules original purpose the decimal module had one purpose and it was not computing decimals the decimal module had one purpose and that was to implement the decimal arithmetic specification exactly no more or no less so that we could port pipe a code from Python back and forth between other languages that respect compliant so should you come to the decimal module and make it easier to use no your varying from the spec that would be a terrible thing to do as a test question should you come to the decimal module and say oh I have a idea for a brand new method you're going toe to toe with a committee that spent 15 years and had people from many companies and many researchers and academia and papers written about what the proper API was the first idea that pops into your mind is probably something they disregarded 15 years ago fair enough so if you come to the decimal module there's lots of ways to want to help that would make it worse really if you come to the decimal module the most helpful thing you could say is you did something here that was not in compliance with the spec in which case we will fix it instantly also I think this is fairly important oh and it applies broadly in the Python world so I want this to be a major message if you want something to be successful you don't have to make something else unsuccessful does that seem obvious it does by human nature is to do exactly the opposite do you think I want Python 3 to be successful I've devoted almost every evening for the past 7 years trying to make Python 3 better do you think I want Python 3 to be successful yeah so in order to achieve that do you think I feel a strong temptation every day to go out and tell people how much Python 2 sucks - there is a urge within the core developer community it was not shared by me and thank goodness cuido stepped on it but a fair number of people wanted to Python - 7 in some way in order to force users to adopt lee force users - we serve users we don't force users users get to make their own choice they want to stay on - 7 they can oh and if they elect to do so we'll support it in perpetuity you know 2020 deadline is will be real if everybody switches over if it doesn't both support it forever you don't have to kill something an example of this in the Python core is the number of people were very excited about concurrent futures they think concurrent futures was the future of concurrency literally and what some of those people did was crippled and closed a lot of the feature requests and bug fixes request tracker items for threads and the multi-processing module like oh that's so old-school though we shouldn't even make those things better because people would want would continue to use them and what but you don't want us to fix a bug as a way to punch somebody in the nose and make them use concurrent futures because you're going to decide for them without knowing their use case in advance what is the right thing for them no if we put it in the standard library once it's in it's our job to make it good even if there's two competing paths of development so this is advice for life but it is all big projects when you get there you're going to champion something and you're going to have an urge to rip something else apart in order to make your thing better I see it all the time we have type simple or simple name space which is kind of cool it's underutilized lots of people don't know about it or like it so the person who was the biggest champion of it went to the name tupple documentation and basically turned all of the name double documentation to what you really want is simple type namespace in other words making that documentation less usable for the people who wanted to actually look up what name tuples actually do and this is human nature there's nothing wrong with that person I'm just asking you to recognize that your nature and to stop doing such things this is advice on how to be usually contribute to open source projects this was the example I was looking at before looking for earlier I knew there was a slide for it here's what the old code was here's what the new code is once you know what this old code does I'm testing this operator to make sure that operator works with sets for a subset test this change is nonsensical this change is checked in along with several thousand others that are just like it so Oh a note on unit tests being sacred I haven't heard these thoughts anywhere else before so I really want to get this out now because it affects you outside of Python world it affects you in your own project why do you have tests one of the great things is as you refactor your code you've got a safety net and you know as you've changed the code that you haven't broken it does that sound useful what you want to refactor code that didn't have a safety net but there is code that doesn't have a safety net what is it it's the test itself which means if somebody sets out to refactor the test the test can possibly stop testing what they were supposed to test for and you would never know so I think this because it has no defense that these are sacred for the most part don't go refactoring people's tests it does not make code better in particular not everybody practises test-driven development I do it once in a while when it's convenient for a particular project but when you do it's awesome because the tests you wrote has a virtue that no other tests had when you wrote it it failed then you wrote code to make it turn green that tells me that the test actually works rock climbers have safety ropes they also put weight on the safety rope to make sure that it's actually going to work a bunch of tests that are written after the fact have attest itself hasn't been tested to make sure that it actually catches anything so in any test written this way or sacred and if you change them later you've undermined something that we will never get back likewise a regression test we had a bug report we put it in a test for it that test failed at one point and then succeeded if you go rearrange a refactor those tests you've mucked it all up and also tests tend to reflect how the author thinks about how the code is supposed to be used it's a pattern of thinking and were you to go rewrite all of those things you've destroyed the pattern of thinking do I highly recommend going to refactor test no go make new tests do not destroy old tests they're sacred ladies and gentlemen thank you so much for inviting me to Pi Bay come join me as a Python core developer you

Channel: SF Python

Views: 43,111

Rating: 4.8968134 out of 5

Keywords: PyBay, 2016, python, django, pants

Id: voXVTjwnn-U

Length: 62min 58sec (3778 seconds)

Published: Sat Sep 17 2016

Reddit Comments

[u/[deleted]]

If one core developer is causing a problem (as he implies), what checks are there to maintain core developer status?

[u/zer01]

Man, Raymond Hettinger is one of those people that I'd love to have dinner/drinks with and just pick his brain. He has a great attitude towards running a team, engineering, and mentoring people.

He's definitely one of my inspirations as a developer.