Type Hints - Guido van Rossum - PyCon 2015

you for that bingo thing yeah can I get a picture with you would for that bingo thing let's do it afterwards oh hello I'm mom sorry I got distracted for a second so I wanted to do this talk in a smaller room somewhere like 710 or so but I asked Diana too late and she said well we have no slots in the schedule and but she said oh there's always some speaker who cancels so we'll stick you in a cancel slot and I said well I don't want such a large audience because this is is this kind of a very high technical topic and I'm afraid that people will sort of not understand it and she said well sure well we'll still find we'll find you a canceled slot and we'll put someone else's name on it and then you'll just sort of show up and so a couple weeks before the conference she emailed me and she said well nobody's cancelled yet and we've got a print that the the program so can we do plan D will will make an extra slot for you at the end of the conference and I still thought oh sure yeah but by then everybody's tired and they're going home or they're already on their flight so I was expecting like 200 people who are really into type theory here but well okay you're all into type theory thank you so much - got this out of the way if you want to see that logo again with somewhere here on the wall so I always like to start with ancient history like grandpa tell again the story about when you found that potato shaped like the baby Jesus so in 2000 which is pretty ancient history for pi from Python there was something called the types a get it actually lived from 98 to 2002 and then it died sad but in 2000 and I found this talk back it still somewhere on Python the dork I had this proposal for type annotations that looks just like what I'm now finally going to propose to put into the language but it didn't go anywhere because there were other proposals and and sort of we didn't understand what it was good for and we had more important things to deal with like unicode so a couple years later in 2004 2005 I started blogging about this stuff again and I guess I had sort of heard about Java generics because I had some little code examples somewhere in the block where I would define the generic function that sort of takes two items of two arguments of type T and it returns a T and I also had this notion of well you can he'll have lists of T or abstract data types like interval of T and that's all exactly like what we're now finally proposing except then I was using parentheses instead of square brackets which is a huge step forward but I didn't know that a little after that sort of I was still very excited about that idea but I couldn't get sort of the right shape for it and and people had all sorts of counter proposals and I thought well okay these these type hints are very complicated and that they need to bake little longer but let's at least put something in the language so that we can give people an opportunity to experiment and 3107 function annotations and so you could actually write all those examples but you would have to sort of define the meaning for that yourself because the function annotations have syntax but now meaning more recently to pythons ago I met a PhD student from Cambridge who was working on actually a Python like language where he was going to do type checking and I talked him into changing the syntax of his language so that it was actually just regular Python all he had to do is come up with the right syntax to boot in those annotation slots and he went off and got his degree and all that and now he's also working with me at Dropbox and then but that that was still fairly sort of interesting curiosity but last summer at Euro Python Bob Eppolito who was also a big name in the Python world for very long gave three recommendations for things that Python could learn from Haskell actually maybe it was from Haskell and Erlang and I forget what the third one was the second one was completely in actionable was something like we should have immutable data types only but the his first recommendation was let's take my PI and run with it and let's do it and so I emailed you Carl let's meet about this let's see what we can do there was a Dropbox hack week other people got interested like Lucas longa and he sort of drafted app app that's just fell in our lap and now here we are and and we're I think we're this close to having a solid proposal that will introduce in Python 3 5 so so what what is this proposal I think that's sort of at the the really high level there are three parts to it there is a separate program which is a type checker for example my PI but there are P for working on competing type checkers and a type checker is like it's something that is more close to a linter than to a Python interpreter so as a developer you get to choose which checker and when to run it and whether you what you do with the output the second leg of the proposal is function annotations there is now a notation for what you put in the annotations that sort of defines what the types of the arguments and the return-type are and how they relate to each other and the third of the third leg of the argue of the proposal which actually is is very pragmatic is that many times you cannot actually put your stubs in your code for various reasons maybe you're not your code maybe it's written in C maybe it needs to be compatible with Python too and so it's also going to be possible to boot start to boot boot type annotations in separate files which we call stub files and more about that little later but why why do we want a static type checker well in an ideal world if you talk to sort of the people who are into static typing and usually they wanted in their compiler but in the end the reason is that static type checks find certain bugs sooner like before you deploy your code which is good if you annotate your code and in the Python world we sort of we have lots of other ways of catching bugs and there are plenty of bugs that don't get caught by type system so this is not a panacea but some bugs actually do get found sooner if you have type annotations and you check against them it turns out that who the larger your project is the more lines of code you have but especially the more people you have working on that project and the longer your project is stretched over time the more these kind of things tools like linters and type checks actually help it helps new engineers plunk old code that sort of only one engineer who knows longer works at the company would still understand otherwise and it turns out that large teams actually already have some form of static analysis running that they found that even without type annotations static analysis can help them find bugs sooner or do other things like yeah well fine bucks sooner sorry that's what it all comes down to right different kinds of bugs and so there are also people who are actually building products but so these static analysis tools currently work okay at least without type-ins why do you want the type ins well python is incredibly dynamic and it turns out that there are many many idioms in Python that actively work against static analysis that just sort of get in the way of static analysis because you sort of you read a string from a file or you asked user for to type a string and then you evaluate it and then you run with whatever object that evaluation returned then type checker has no way to figure out what that type is and then all the rest of the code that uses that value also doesn't know what the type is so often there are ways to to certif helped help type checkers to keep track of what what types are actually in used it in the insert of in the programmers mind that's often what it what it cares well it what it matters what matters most and so it turns out that because type hints are important to represent what's in the programmers mind they're also quite useful as documentation and yes they get out of date but that's where the static type checker helps the static type checker actually checks whether the type the type annotations that you put in your program match the code that you put in your program which is big advantage over putting the type information in docstrings because nobody checks that the dark strings are still correct it also turns out that everybody is building IDs that have all sorts of handy stuff in them and it's very useful for an IDE to be able to know what the type of a particular variable or argument is so that when you hit the dot and then you ask the IDE what could I type here that starts with an A and it can know that if it's a list that append is a pretty popular choice but if it's not the list append is probably not one of the choices IDs also like to put squiggly red or green lines under your code to help you sort of catch type errors early so all these things oh yeah let me quote a statistic someone who works for one of the big Python IDE developers mentioned at the language summit that they can correctly infer the type of about 50 to 60 percent of all expressions occurring in typical code which means that nearly half of the expressions they cannot infer the type and that's where the type hints would be useful so why do you want these stub files why do you not always want to put your types your type in C in your annotations well the original reason why stop files were invented was that you somehow have to annotate the standard library and much of the standard library especially the built-ins is all implemented in C and you can't expect the poor type checker to also parse the C code and understand what's going on there because of this in the C code everything is an object how great it turns out there are many other useful use cases where stubs are handy like third-party packages you don't really want to sort of go modify the source code of a third party package because you would have to do it again for the next release and there are all sorts of reasons there's legacy code where there are places where that code is used where the type-ins would be in the way like for example Python 2 compatibility there are also reasons that maybe maybe the owner of the code just doesn't want you to put the type hints in there are plenty of people who don't like type hints and they don't want you to mess with their code but if you are using their package you might still want to sort of behind their back or with sort of with at least without their explicit agreement create a stub file that describes the types of their particular module or package so that you can type check your own code better plus it takes a long time to develop stubs and we don't want to wait until we have stubs for the whole standard library so we hope that actually people will start creating more stubs for more standard libraries modules after three-point-five has already been released and yet a lot of people when you sort of present this this idea to them there they're not just lukewarm they're actively resentful Python is dynamically typed we don't want type chat we don't want to have to write types in our programs well correct you don't have to do this I'm just adding this so that those people who have large code bases where they think that they can use type hints and that there that will make their static analysis better can use it and it's it's useful to have a standard notation so that everybody can sort of share stubs and agree on what those stubs mean but very much please please understand this is optional I'm not telling you you have to put stubs and I'm not even asking you to put I'm not even asking politely to put stubs in I'm telling you there is a proposal for stubs sorry for type type hints use it if you think you'll benefit from it ignore it if you don't care you can run the type checker or you can choose now to run a type checker that's completely up to you also in Python 3.5 we're making very sure that it's not gonna break anybody's code and it's entirely provisional which which actually means provisional in this case means that we can still tweak the API in certain backward incompatible ways if we find that we made a mistake and that's also important because we don't we don't want to sort of paint ourselves completely into a corner but we do want to release something at 3/5 so I think I already said most of this currently a couple of companies have their own sort of notion of stubs that they they maintain themselves and I think it's just a good idea to have a PAP and accept that PAP of course so that everyone can say ok the issue of the syntax of the annotations and stubs is settled now we can all compete on who has the best type checker or the best type inference err or the best IDE and some people like type systems so if you were to go back to those old blog posts and that type cig proposals in 2000 you'll find that there was a lot of Hope where we thought oh if we put the type annotations in with the compiler can generate more efficient code and our code will magically run faster well on the one hand if you want your code magically to run faster you should probably just use pi pi which works just fine without type-ins on the other hand there are certain systems like siphon that can perhaps use type hints just as they are standardized or maybe they'll have their own idea and then we'll have to talk more but well we'll see how this works out but I'm not yet holding my breath for faster code due to type hints however I am expecting fewer bugs and happier programmers due to type hints so again some of you have probably read PAP 3107 or the corresponding documentation and gone to town defining your own type checking system or maybe command-line parsing system based on the type the function annotation syntax and your notation looks nothing like that 484 well the best I can promise is that is totally in your right your code will still work in 3.5 it may not please a type checker but you don't have to run the type checker on that particular file or module or package if you want to peacefully coexist with type checkers that PAP actually also has a standard notation for turning off type checking and the simplest form is a decorator that you can put on an individual function so in much of the rest of the talk I'm going to try and explain what these type hints actually look like if they're not as simple as this is an INT and ethicist string this is even a tiny bit of theory but I get confused by the theory myself sometimes and it ends with a bunch of stuff that is necessary if you actually want to use this in practice so basics of what I what we call gradual typing it's gradual because you can have tie pins in some code and no tie pins in other code and something useful needs to have when annotated code and unallocated code needs and the idea is that the annotated code must conform to its own type hints so if you if you say this is an INT and that is an INT and then you do something with them that operation that you do on those two arguments better be something you can actually do with inns otherwise the type checker will be unhappy and tell you on the other hand the code that you didn't annotate the type checker is going to be silent because that's dynamic code and sort of by by the definition of the idea of gradual typing an annotated code is not checked and there is actually a special type in the type hinting system named NE which means shut up and not annotating a function is almost the same as annotating it with any all-around technically that's not entirely true so don't help me too much to it but that's sort of a first approximation interpretation of gradual typing so this anything is actually a pretty strange duckling and I don't know in which it's sort of how you draw your your class diagram so I don't know what's at the top or what's at the bottom but I can tell you that any is both at the top and at the bottom of your class tree because that's the definition of any and so it's let's say that object is at the top of your class tree and everything that derives from object and everything is below that then any is at the top because any value X is an instance of the type any and any class C is a subclass of the class any however and this is completely I mean so this first bullet any behaves just like object because all those things are true if you use object instead of any however because any is also at the other side of your class tweak any is actually a subclass of every other class and that is that is the magic of any that it also actually causes some trouble with is subclass if you take it literally because normally is subclass is transitive is if a is a subclass of B and B is a subclass of C then we know that the a is a subclass of C but if if something in the middle of a chain like that is any then you can't sort of go through that any because otherwise you'd end up with every subclass every class is subclass of every other subclass and that would be very useless type system so technically we're not using we shouldn't be saying subclass we should say is consistent with and this terminology I believe is due to Jeremy's seek of Indiana University and if you just google for what is gradual typing you'll find his excellent blog post which also has a few good things to say about dynamic versus static typing in general but there is a little definition where you say some type is consistent with another type which essentially means values of the first type can be assigned to variables of the other second type and this relationship is not in general symmetric or transitive but it is mostly that because what follows from being consistent with is the regular subclass property however is consistent with also works for any in the opposite direction so any is consistent with T and T is consistent with any and this is true for every T but you cannot conclude from this that every T is consistent with every other T because the relationship is defined as not transitive and that's as much theory as I can manage to try to explain but read the blog post is much better so back to sort of practical stuff there is actually in the whole proposal there's really only one thing I'm proposing to actually put in the next C Pyfrom distribution as part of the C Python standard library and that's the typing module there will be no type checking as part of C Python there will be no new syntax for typing every syntax I've shown you so far is actually already valid Python syntactically all you have to do is overload gather item in a few odd places we will also not put annotations in any other standard library modules and we may sort of in three point six we may change your mind about that but 43.5 typing the PI is the only thing we want in the standard library and thereby sort of the notion that there is now a standard notation which is defined by what's in typing the PI and what's in the pepper so if you want to use any or a number of other magical objects like unions or dictionaries and lists that can be sub scripted with types you have to import them from typing however that doesn't mean that you necessarily always need to start by import typing here's a simple class example that well it's pretty silly class I guess it's a chart in class with you in the interface defined by someone who had never defined a class before so there's this it would actually I wrote it set a label takes two arguments that are floats and one that's a string in it returns a bool maybe that that tells you whether it actually succeeded in setting a label and apparently there's a get nearest method that asks for the nearest label to a particular point on the chart there are also a couple of helper functions those those are not part of the class they're not methods they're just functions these are that they don't have a self argument as you can tell they are annotated make label takes a chart object so we see that user-defined classes can also show up in type in positions and well it does it actually calls the set label function the type checker can figure out that this is actually correct code that set label a is correct because set label takes a string and so on get labels so it takes a list of points and it just says that the points variable is a list that's a built-in list type and it also tells us that it returns a list which is all very partial information because actually if you look at the code it calls get nearest for two points so apparently the list of points is actually list of tuples of two floating point values and so what it returns is list of strings but you can't really tell that from the code here and the type checker would not be able to validate that you didn't make a mistake with that so here is a little bit of extra stuff that you can add we're going to import the capital list and capital to pool from the typing module and we're going to use those I don't have a pointer that's okay so we have now weari declare the get labels function as taking a list of tuples where they're actually tuples of two floating points and it returns a list of strings now that function doesn't actually care that the input is a list it can work with any iterable so let's import iterable from the typing module and then we can say it's an iterable of tuples and these are all actually working examples so but what exactly did we do here well typing dot iterable we set from typing import hits are also there so something to find in typing named iterable which is almost the same thing as the collections ABC named interval and that particular collection pretty much defines one piece of behavior which is that it has an under under hitter method and so the typing dot iterable does the same thing but it also lets you specify what's that that it'll actually contains or returns for various pragmatic reasons mostly having to do with not wanting to modify the cpython implementation at all typing that list is the moral equivalent of the built-ins list function or type actually but again you can specify what kind of items well the type of the items is tuple again represents or resembles the built in tuple function class typing the tuple is actually a little bit more special because we usually don't want to think of tuples as a variable immutable sequence of X's we probably think of the typical tuple alike in this example of it's a to tuple of two floats or maybe it's a tuple of two integers and a string and so that's what you can define with the tuple class now maybe in the future we can just write lowercase lists but for now in three point five in this proposal because we want to have just the single typing module that works in Python three five but actually also in previous versions you have to import that thing from the from typing oh and the way all these things work is there is a meta class that overloads getitem but you don't really have to know about that that's just sort of if you're surprised that you take a class and you put square brackets behind it if you're surprised that that's valid Python that previous slide was why but well there's there's there's a sad terminological problem in Python which I am personally responsible for which is that we've always done a terrible job of distinguishing between types and classes and for example the function to get the class of an object is actually named type and well they're historical reasons but for if you're talking about type checking and type hints it actually makes a little bit more sense to be careful about your terminology so in cases where it actually matters we will say class when we meet a concrete implementation and type when we mean something that is more in the mind of the type checker this again is pretty subtle and we may have to iterate on this terminology but this is what it is currently anyway so here is I believe a complete list of things that you can use in type hints and that type checkers are expected to understand so you can put classes in there could be built-in classes like object or float or could be user-defined classes like chart either in your own code or in some third party package then there are these generic types which are things like lists square bracket int or a dictionary with a string and an int where the string is the key in the interests of value abstract classes like interval then there are a bunch of magic things like ne and Union and it turns out that tuple and callable are also pretty magic and then there is a way of defining your own generic types do-it-yourself generic types you know show that later hopefully they'll all have time for that so Union is a very basic type that says it could either be this or that type so for example the union of int float and string those are all things that you can add to them one very common case is union of something and none actually something are the type of none because the arguments for union should be types we let you actually specify none also but either an int or none or either a chart or none or either something or not not something is so common that we have a special notation optional and that sort of that helps the human reader probably more than it helps the type checker because the type checker just reduces that to a union and works with that so I mentioned that tuple was a little bit special tuple actually has a variable number of parameters and sort of each of those is describes a different position in the tuple so tuple of int instr is a tuple of length 3 where the first two items are in sin the last is a stream of course we got a lot of pushback and was an endless thread about well but is a tuple not really just immutable sequence and well often we use a tuple as in an immutable sequence so as a sort of compromise you can say tuple of float comma dot dot dot and that's literally three dots and that is also already existing Python syntax I think way back since in Python 2 that was introduced to help the numerical Python people with certain types of slices so that is what you would write if you meant a tuple really as an immutable sequence of floats more stuff again this is this has been a fairly controversial way of of naming callable some people wanted to call it function but not all color balls are functions there have been different proposals for how to separate the argument list from the return type in the end I looked at all the different proposals and this still came out as that sort of the most straightforward and Lee or prone so you have a callable that takes three arguments these are the types and it has a certain return type if you have a callable that takes like keyword arguments or as varargs or has other strange things currently the pep for aid for notation cannot actually describe that because the proposals for describing all the possible signatures that you could have in Python def syntax got very complicated because the options are that the possibilities are endless we didn't want to to sort of color ball is really usually used for very simple callbacks that are almost always called with a fixed number of positional parameters and nothing else if you have a callback that is more complicated than that you can say three dots instead of the argument list and then the type checker is just supposed to take a break from that argument list generic types how do you make your own generic types you have to define type variables that's little clunky in Java that type variables are introduced by special syntax but in Python the syntax is not all that special so you have to tell Python I'm defining a type variable but usually most people never get to the point where they actually need to define a generic class because this is usually only needed when you're defining a new container type if you are defining a new container type you may have to define one or two type variables in an entire module so it's not much overhead to be forced to actually use that type fire function to define those type variables and now we can say we have class chart and apparently this is a variation of the chart I had before where the labels could be of type T and so we can have a chart of string but we could also have a chart of a tuple of int and string which would make every label a tuple of an int in the string of course nevertheless this is just sort of a watered down for you know things you can do in Java you can also do this with functions you can define a generic function without having it inside a generic class I don't really care about this detail all that much I also have a request for the organizers can I go a little over thank you I will I will try not to make it too crazy here is here is one particular example I'd like to show let's say we have a function that calls the built-in split and we try to annotate it and it takes a line which is a string and a separator which is an optional string essentially and it returns a list of strings and what the function actually does is it splits the line but it limits the split to at most one split point it turns out that that split method also works for bytes and we would like to express the type of this function so that it's legal for the color to call split one with a string or with bytes this is Python free of cars without strings and bytes or different things so our first attempt and bear with me let's say that any stir is a union of string or bytes which means that can either be a string or byte by the way this this is valid you can define your own type aliases so that you don't have to type so much so we define our split one function and now we say the argument is in any stir and the separator is an optional and easter and it returns a list of any stirrers and the body is still the same sadly there are two problems with that the first problem is that the function is actually more strict than what you would think given that signature because if the line is a string and the separator is a bytes it doesn't actually work and it turns out that if you feed that to the type checker the type checker is full of enthusiastic rejection of what you just wrote so we do something else we use a type variable and we this is a little bit of magic that you can actually came up with we constrain the type variable by calling type far with still type var always needs to take the name of the type variable that you're creating but now you give it extra positional arguments that are themselves types although they look like classes and now this type variable works like a type variable n Easter is just like that T we had earlier but the type checker also knows that when you call it so it knows it knows basically two things it knows that the line and step arguments must correspond they must either be both strings or both bytes and there must not be anything else and so that's how we solve this particular problem and it turns out okay if if we now make the runtime mistake we get a complaint from the type checker but the function itself is perfectly fine according to the type checker which is correct it is perfectly fine and this is actually such a common case that we have any ster predefined exactly as I was written on the previous slides in the typing module the constraint is a little bit magical that if any of the arguments is a sub has a type that is actually a subclass of ster that the type the type checker reduces that to exactly the the base class that is actually listed in the type variable so that we don't make the mistake of expecting that it will also return that same subclass because often subclasses of ster are very impure and their operations return regular stores I think that was the most complicated part that I wanted to talk about there is a whole bunch of pragmatic stuff like if you have if you're defining a container type or say a node for a binary tree you often need to name you want to mention the class that you're defining in the argument signature for some of the methods and it turns out that the way cpython works or the way python is defined that class object is not actually created and the name node is not actually defined until you've reached the bottom of the class so the reference referencing the class in the annotation of one of the methods doesn't work and as a compromise you can put string quotes around it and the type checker will just strip the string quotes and parse it that way well see Python or pipe I will just execute it as is and set the string in the annotations variable lots of other pragmatic things like in some cases you want to annotate variables we don't want to introduce yet we don't yet want to introduce a syntax for annotating variables in Python 3 5 so as a compromise again you can put that in a type comment there's a cast function there's some debate about what the order of the arguments should be I think I'll just make it the first there are some other things like undefined I also implemented in the typing module instance and class checking so that if you actually have a a union of int and ster you can actually ask is 42 an instance of that and it will say yes and if you ask if 3.15 is an instance of that it will say no stop files oh yeah last minute this sort of choice was that we want this type files to have a different extension that wasn't clear in the early stages of the proposal but it is just too handy to be able to have stop file live in the same directory as the source that it describes even though that's not always how it works and so stop files are named py hi what'd you put in the stop file is simply class definitions method definitions function definitions but you can leave the bodies of the functions as short as you can like just put paths everywhere because the type checker looks for the signatures but it doesn't do any type checking otherwise overloading at some point we would like to have multiple dispatch but a good multiple dispatch implementation is really hard and we kind of ran out of time and we don't want to wait for the multiple dispatch implementation to land before we can use it in stubs because they understand especially amongst the built-in types and functions there are a couple of things that really are sort of in dire need of having an overloaded signature this example is from the bytes class if you pass got get item on a bytes object an integer argument and it returns an integer it returns the byte at that position but if you pass it a slice it returns a bytes object which is the slice of of bytes indicated by that slice so that's a clear example of overloading but sort of as a compromise you can only use overload in stub files let's see what else do we have oh yeah and I mentioned this earlier if you're if you're already a happy user of type annotations and you have different interpretation of type annotations you might want to disable them so you can stick a class or method decorator know type check or you can disable it for the whole file by putting type ignore comments at the top you can also probably use a stub file because the type checker will also prefer always prefer the stub file over the implement let's see what else is there to discuss there were a whole bunch of rejected alternatives the reason I ended up with the current somewhat clunky syntax that you can look up in pep for 8:4 is that it's easy to parse for example the angular bracket syntax is notoriously hard to parse because angular brackets are also used for comparison operators and in Python sort of there is no different portion of the syntax where types are used types are just possible everywhere because their classes traditionally are just objects the current notation has the advantage that there is no new syntax added so the typing module can actually be used back to Python fleet of 2 at least we didn't need any changes to C code we didn't need to make any changes to other standard library modules and it sort of its uniform when you see a capitalized word followed by square brackets you can be pretty sure that we're talking about some kind of type thingy now if if if you're one of the 2,000 people in this room who has a strong opinion about type systems there are probably a few things you don't like about this proposal well it's not perfect but it will still help you find bugs what can I say so how is it this PAP going to be accepted I don't know it's probably I I'm hoping to find someone I have someone in mind to but not myself not one of the PAP offers that's that sort of a big constraint who is sufficiently independent to give lots of good pushback on this proposal and that would be the ideal be DFL delegate for this PAP if you're not familiar with that notion in general when there is a pap and I don't understand the topic or I'm too closely involved with the creation of the pep someone else some other senior person or knowledgeable person in the Python community needs to sort of volunteer to be the BD FL for that particular pep so they can have the last say in approving or rejecting it again it will be provisional which means that we can still tweak it while 3.5 has already rolled out and with apologies to the organizers I don't know if there is time for questions or if we should just go to the closing remarks no questions but come track me down I will I'll be right here

Channel: PyCon 2015

Views: 45,593

Keywords:

Id: 2wDvzy6Hgxg

Length: 49min 26sec (2966 seconds)

Published: Sun Apr 12 2015

Reddit Comments

[u/tavert]

Dunno, the "not for code generation" bit and the "punting on dispatch, it's not ready yet" bit mean there may be a superficial resemblance, but the purpose and practical implications are quite a bit different.