"Python Oddities Explained" - Trey Hunner (PyCon AU 2019)

okay Trey Hana helps Python developers level up their skills through his weekly Python skill building service Python morsels and through team Python training sessions today Trey will take us through a number of pythons unique features and quirks and attempt to reshape our mental models of how Python works please welcome Trey Hana to the podium [Applause] all right is everyone awake so this talk is a little bit longer than it should be so I'm gonna speak quickly it's also confusing so I'm sorry my name is Trey and I help folks level up their Python skills one of the ways I do that as was said is through Python morsels a weekly Python skill building service I send that one Python exercise every week there the other way is through team training so I do on-site trainings for teams too and I help experienced programmers become experienced Python programmers when I am doing on-site trainings I often get to see some very weird code because my students are playing with Python which is lovely because playing is a wonderful way to learn sometimes though while I'm trying to explain what pythons doing with some weird code I realized that I don't really understand what pythons doing at least not well enough to explain it to someone else so this talk was inspired by all of the people who asked me weird and confusing questions while I was at your company teaching Python thank you for your confusion this is a Python 3 only talk if this talk was a Python 2 oddities talk it would be even longer and much funnier if you're interested feel free to ask me about Python two oddities in the hallway later so we're gonna get started a quick warning again this will be fast I'm sorry so first we're going to talk about variables let's say we have a variable X that is equal to zero and a variable numbers that point to a list of numbers if we loop over this list of numbers and assign Y to the square of each of these numbers what will we see after this loop when we look at Y what is your guess feel free to shout things out I'll remind you again you're allowed to shout things out in this moment in particular zero zero is the guest okay so we see not 0 64 what's your guest that we'll see when we access X we see 8 we see 8 because so what what happened here is variables that are assigned inside of a loop are also accessible outside of that loop so that for X in numbers that is actually an assignment that happens each time that we iterate through the loop the same way that y equals x squared is an assignment so that in that for X in numbers is the same thing that happens is an equal sign in a for loop let's look at the same thing we're gonna take the same x equals 0 the same numbers list we're going to make a comprehension representing the same list of squares there what do you think X will be this time what's your guess we got 0 we got 8 who thinks 0 who thinks 8 who thinks something else ok I don't know what you think it is but I'm curious so X is actually still 0 here unlike with for loop so in Python 3 this is the case in Python 2 this is the only Python 2 oddity Python 2 used to leak scope in comprehensions the same way for loops leak their scope let's say we have a global variable numbers and we want to make a function that uses + equals to add more numbers to this list what will this do what's your guess so when we have add numbers with the list 4 or 5 6 this could modify numbers it could instead make a local variable numbers could give us an error so I'm gonna let you think about this for a moment this gives us an error so plus equals it reads from and assigns to the variable that's on its left hand side that numbers thing there so we're trying to read from a global variable and write to a global variable which Python doesn't like it doesn't like it not because the plus equals the plus equals isn't the weirded E or the oddity here the weird e that's a new word so if instead of using plus equals we use an equals and a plus kind of the same thing we will still get an error when we defined this function Python parsed all of the assignments all of these something equals something then claimed each of those variables to be a local variable it painted them to be only local variables since numbers is a local variable we can't read from it before we write to it that's impossible we've got a right to a local variable for we read from it and so python gives us an error here all right one more example the same thing if we make a similar function that prints the global variable numbers and then afterwards assigns to it if we call this function we will also get an error the variable names in Python cannot be both local and global in the same function so we're allowed to read from a global variable we're allowed to write to a local variable we cannot write to a local variable that has the or rather write to a global variable and writing to a local variable that has the same name as a global is just confusing so Python doesn't allow you to do this it doesn't allow you to read from a global variable and later write to a local variable of the same name because that would probably be a bug in your code so this is probably hopefully something you've never encountered if you have encountered this you probably have some very weird code but the moral here is that we're a lead we're allowed to read from a global variable without writing to it but once we've assigned to a variable it is only a local variable which means that if we swap these two lines of code our function doesn't give us an error this time the function prints out the expected numbers what our global variable numbers hasn't changed because numbers is a local variable now because whenever you do an assignment you are assigning to a local variable not a global variable assignments in Python only change local variables so what if instead of doing an assignment in add numbers we use the extend method is this any different what do you think Python do in this case so this time it could print out one two three four five six it could give us an error who thinks that it's going to print out one two three four five six who thinks that it will give us an error so Python if we pass it and one two if we pass in four or five six you're just guessing one two three four five six even though I didn't have the code up it doesn't give us an error it does print out one two three four five six the tricky thing here is that and in fact it actually does mutate numbers there the tricky thing here is that we're changing numbers but not the same way we changed it before we can't assign to a global variable I said that before we're still not assigning to a global variable though we are mutating an object so there are two different ways of changing things in Python you can change a variable which is an assignment you can change an object which is which is a mutation we're doing the second one of those we're not doing the first one of those we can't assign to a global variable but we can mutate any object that we can get our hands on it doesn't matter where it is as long as it is a mutable object so you're allowed to read from global variables in Python but you can only write to local variables and every assignment statement writes to a local variable there are some escape hatches in Python to get around this you can technically write to global variables I'm not going to show you how to and you shouldn't do it list comprehensions have their own scope in Python which is great you don't really have to worry about that much and because of for loops though do not have their own scope which can be a little bit tricky and the reason for that is Python is not block level scope so if you're from a programming language with curly braces it might work differently from Python depending on which one it is and in general you don't need to worry about scope if mutating an object if you're just mutating there's no assignment happening scope rules go out the window scope doesn't matter mutation matters which leads us to our next topic which is mutability assignment is about changing variables mutability is about changing objects let's say we have again a list of numbers and we assign numbers to two numbers if we append two numbers to how many elements do you think are in numbers two right now what's your guest so we've got one two three or four so there are four things and numbers - not surprising we append it to numbers - there's a fourth thing in there what is your guess how many things are in numbers 1 2 3 or 4 also 4 there are 4 things in both of these because we are referencing the same object with two different variable names mutating one of these lists also mutates the other one because they're the same list - names that point to the same object changing objects and changing variables or two distinct things in Python assignments change variables by pointing them to a new object mutations change the object itself which any number of variables could be pointing to can we make a tuple with a list inside it is as possible what's your guess yeah we can python allows us to do this what if we call the append method on the first list in this tuple apply then allow this does allow this doesn't give us an error and this tuple now has two things inside that first list they're tuples are immutable meaning that can we can't change tuples but we haven't technically changed the tuple at least not from pythons perspective we changed a list that just happened to be inside of a tuple so if we take the first list in this - pond we were to assign it to a new variable and then append to that new variable we'll have changed the list everywhere that it's referenced the same way this is probably a little bit less confusing if you were confused by that first one the trick here is that tuples don't actually contain objects they contain references to objects variables in Python and data structures do not contain objects they contain pointers pointers to objects what will happen if we make a new list an empty list and then try to append that list to itself does Python allow this who things will get an error here this doesn't make any sense other things we're gonna get an empty list a list with an empty list inside it something else so it is something else Python represents it helpfully this way I say hopefully because Python didn't want to blow up our interpreter here by showing us and infinitely recursive data structure so Python allows us to do this because this is kind of the ultimate proof that lists do not contain objects they contain references to objects so the list is simply referring to itself which is something you can do at Python not something you should do in Python but it is valid Python code nonetheless so lists are not buckets which contain objects instead they're kind of like look-up tables which reference memory locations where those objects are stored so data structures lists tuples dictionaries they don't contain objects they contain references to objects variables also don't contain objects they just point to objects which means that change is an ambiguous word in Python you can change a variable by assigning but that's not going to change any objects and it's not going to creating new objects you can instead mutate an object which changes objects but it's not going to change any variables it's not going to point to new objects or create new objects if you feel rusty on this topic if the ID of this was confusing there are two talks that I would recommend watching one of them is facts and myths about Python names and values by Ned Batchelder the other one is names objects and plummeting from the cliff by Brandon Rhodes who gave that great talk yesterday on the entities alright so let's talk about ducks let's say we have a list called duck list what do you think will happen if we use the plus equals operator to add a tuple of two values this list will this work well this worker give us an heir who thinks this will this will work not gonna give us an error who thinks we're gonna get an error here so we actually don't get an error pie that allows us to do this what it does is it loops over this tuple on an edge each of the items from that tuple into our list let's make a duck tuple that has two strings in it and we're going to do the same thing but we'll use plus equals to add a list to this tuple who thinks we're good an error in this case I think we're not get an error so of course everyone thought we'll get an error because it's an oddity something must be going on different here we do get an error python tells us that we can't use plus equals to concatenate a list to a tuple even though we just use plus equals to concatenate a tuple to a list the error here isn't because we can't use plus equals on tuples we're allowed to use plus equals on tuples python is fine with that the error is that we can't use plus equals on a tuple with anything that isn't a tuple that's where python gets upset so plus Eagles on tuples only works with other tuples tuples aren't the odd one here lists are the odd one so we're gonna take an empty list and try to do plus equals with a string will this work over this given it's an error who thinks it work who thinks we will get an error so this does work what we get though might not be what you're expecting it loops over the string it puts each of the characters into that list because what it's expecting is an iterable anything that you can loop over and when you loop over strings you get character strings so plus equals on list does the same thing as the extend method on lists the extend method also accepts any interval loops over it and appends each of the items that were in that interval to that list so plus equals is consistent but not consistent with plus equals on tuples is consistent with the extend method on lists the list extend and plus equals situation this isn't really an isolated issue many other things in Python accept any iterable under the Sun so for example the string join method accepts lists but it also accepts tuples and it even accepts strings what happens what do you get when you / strings yeah characters we encourage as we loop over them spaces in between those characters likewise this is kind of a weird one the dict constructor accepts any to item interval it will give you the first item in that interval as keys the second item has values which means we get a dictionary like this which again is something you can do in Python not something you should do in Python because it's going to confuse your coworkers in all these cases though Python isn't type checking there's no type checking going on it's assuming that you gave it an iterable that's sensible what it's doing is it's practicing duck typing the idea behind duck typing is if we need to identify whether something is a duck we don't check the DNA to see whether it's a duck we instead say if it looks like a duck and quacks like a duck it's a duck that means checking the behavior of an object instead of checking what that object actually is what class that object is so we don't usually care whether something is a list we care whether it's a sequence or maybe whether it's an iterable and we don't care what if something is a function we care whether it's a callable whether we can call that thing we tend to use generic words in Python to describe the behaviors of things more than we tend to discuss the types of those objects then I just forgot to hit the arrow key over and over here so takeaways the list extend method accepts any interval the plus equals operator works the same way as the extend method on lists on tuples the plus equals operator and most other types that are immutable that plus equals operator does not work the same way on list it doesn't accept any interval it only accepts that same type of object and in general in many cases Python like with that plus equals on list doesn't do type checking Python tends to attempt a certain operation and hope for the best if we in our own code embrace duck typing by thinking in terms of behaviors instead of in terms of so thinking in terms of intervals instead of lists for example this will oftentimes make our Python code it'll help us understand what Python is doing but it'll make our Python code better because our Python code will be more like Python itself it'll kind of blend into the ecosystem of Python so I'd recommend familiarizing yourself with terms like iterable callable hashable sequence mapping these kind of generic terms that describe behaviors more than they describe types and if you're interested in how iterables work in particular there is a talk that you can check out that I gave at PyCon Australia two years ago called Luke better a deeper look at iteration that's specifically on just a deep dive into intervals all right we are going to take a quick mental break before this last section which is going to be just as confusing as all sections before this so here is a dog in a mailbox a chipmunk drinking some tea and a cat debugging its code all right that's enough of a break we're going to look at that plus equals operator again real quick so we're going to take two variables and assign them to the same tuple so a equals B equals a tuple one two we're now going to try to use the plus equals operator to modify one of these tuples does python allow plus equals on tuples it does we just saw this so Python allows plus equals on tuples we can use that plus equals operator works just fine what is B at this point so a had one two three four it had four things does B have two things or four things what's your guess it has two things B as two things a has four things so plus equals on tuples you can think of it as the same as a plus with an equals we're not mutating this tuple we're just making a new tuple and reassigning the same variable name to it so X plus equals something is the same as x equals x plus something both of these statements make a new object and then point our old variable name to that new object all right we're going to take two variables assign them to the same list and then we want to use the plus equals operator to modify the first list what will the second to list be at this point we'll have two things or four things what's your guess it has four things had two things with tuples has four things with lists the plus equals operator doesn't mutate tuples because it can't but it does mutate lists now I find this interesting because this means that on lists a plus equals something does a mutation but a equals a plus something creates a new object and then reassigns the variable to it so in Python a plus equals B is not always the same as a equals a plus B these do the same thing on numbers on strings on tuples and another immutable types they don't do the same thing on lists the reason for this is that lists are mutable and the plus equals operator is called an in-place addition operation meaning it's supposed to perform a plus operation in place so it's actually supposed to mutate the object we're working with that's technically the expected behavior if it can if it is a mutable object but it's convenient to use plus equals on tuples on strings on numbers and things that are not mutable in Python Python cannot mutate these things so what it does is it falls back to regular old addition followed by an assignment so on immutable types in place operations return a new object to us but on mutable types in place operations return the same object back to us and they mutate that object so we're gonna take a tuple with a list inside it we've already seen that we can do this this is valid in Python we can store a list inside it to believe and then the list is mutable the tuple is not what will happen if we say X sub 0 plus equals 3 for what's your guest so it could mutate the tuple it could give us an error will rather not mutate the tuple it could mutate the list that happens to be inside that to plug could give us an error so who thinks that it will mutate that list inside the tuple who thinks that it will give us an error alright so that was almost everyone thinks it will mutate the to only one error fortunately though none of you are wrong this a gun does give us an error and it does also mutate the list so we're all happy this is a great compromise this is a very obscure quirk of the way python does plus equals it's never going to bite you in production code but we're gonna take a look at it anyway so plus equals what this does is when you use the plus equals operation whenever you use plus equals Python calls the dunder I add method on the object that we're pointing it to which in this case is a list after that it does an assignment so it does dunder I at first then it does an assignment so the exception happens because the assignment fails because we can't assign into a tuple but this happens after dunder I had successfully mutates our list so the list dunder I add method mutates our list and then returns the same list back to us it returns itself back to us for the sake of the assignment operation but since we're not allowed to assign it to tuples we get an error so this is never gonna bite you in real production code because you're never going to use plus equals to mutate a list that happens to be inside of a tuple and expect anything good to happen but this is an interesting excuse to learn about how plus equals actually works on the hood nonetheless these kind of dunder methods that power Python so some takeaways here the plus equals operator always performs and assignment but plus equals and other augmented assignment operations that's what those assignments that also do an operation are called they are allowed to mutate the objects that they are called with if the object chooses to mutate itself lists just happened to be the first mutable type that you encounter in Python I'm the one that you most see something like plus equals mutate on but tuples don't do this because they are immutable neither do strings or numbers or the other immutable types so the reason for this is according to the Python documentation in place additions are supposed to do their operation in place whenever it's possible to do so if Python doesn't allow you to do that in place operation because it is immutable it is supposed to fall back to a plus with that equal sign if you find in your Python code something that seems like a bug in Python it doesn't necessarily mean you found a bug in python it might mean that your assumption of what python is doing is different from what python is actually doing from the way that it sees the world so I have some closing thoughts from for you regardless of what programming language you're in whether it's Python or some other programming language day to day it's important to understand how the fundamentals of your programming language are different from what you'd expect either from other programming languages from a math background or from something else so how do function calls work how do variables work how do objects work it's also important to understand the way that your programming language thinks about the world what does the world look like through the eyes of the Python interpreter and if you think you've found a bug in Python it might just be that you miss understanding what trade-offs the Python core developers took when they were implementing a certain bit of functionality that you're seeing so if you see something odd going on in your code before you fix it poke around and see whether you can learn from it breaking things is one of the best ways to learn and if you find a Python oddity of your own and you tweet it out use the hash tag Python oddity because there are dozens of other things there that I've tweeted I have last few years along with the number of other folks and please do share these if you find them I love finding these they tend to get fixed if they're actual bugs so some of the old tweets are a little bit boring but tweet them out if you do find them that's all I got for you thank you you [Applause]

Channel: PyCon AU

Views: 6,304

Keywords: pyconau, pyconau2019, Python, PyCon, PyConAU, TreyHunner

Id: 4MCT4WLf7Ac

Length: 26min 41sec (1601 seconds)

Published: Sun Aug 04 2019