Python Context Managers and the "with" Statement (__enter__ & __exit__)

[Music] hey there this is Dan and today I'm going to talk about the width statement in Python so the width statement relates to something that is called context managers and it is sometimes regarded as a bit of an obscure feature by some people but actually when you peek behind the scenes there's relatively little magic involved and it's actually a highly useful feature that can help you write cleaner and easier to understand easier to maintain code so the first question that I want to talk about is okay so what is the width statement actually good for the width statement is great for simplifying some of the common resource management patterns that you'll see in your code so by resources in this case I mean system resources like files locks or some kind of network connection stuff like that and so with the width statement you can abstract the way some of the functionality there that is you know commonly required like dealing with the acquisition and releasing these resources again so a good example here would be dealing with files a file and Open File is a system resource and you need to acquire it in some way and then also release it to give that resource back to the system so it doesn't have to keep track of the file you're working with and so I want to take a look at this simple example here where we're using the with statement to open a file and in write to it and to have the file automatically closed so the big advantage here is you know what I just said where if we use the with statement to open the file Python is going to make sure that this file is going to be closed once we're done with it so basically when you do this you open or you create this context where we're working with the file object right we're giving it the name F just maybe not an ideal name right maybe you should call this out file or something like that and then in the block that follows in this indented block here we can work with F and as soon as the execution of your program leaves the context Python will automatically close the file so it's going to call F dot close and that will return the resource the file descriptor back to the operating system and it doesn't have to keep track of that resource anymore and so to demystify how the width statement works let's take a look at what actually happens behind the scenes right so if you were to write this manually and probably do something like that right so you would open the file just assign it to F here on this line and then I'm using a try finally statement here to actually write to the file and then to make sure I actually close the file so even if something goes wrong here you know we have some kind of exception some error where we can't write to the file this would make sure that we're actually calling clothes on the file which in a longer running program that's super important because you always want to give up these resources so the try finally here is really significant right it wouldn't be enough to just say okay I'm going to do an open and then write and then a closed because you could potentially leak that resource by not releasing it again if some kind of exception would happen using the with statement really simplifies the code we have to write here because what you see in the second code example that is super common you're going to use a pattern like this very very frequently if you're dealing with any kind of resource like open files open network connections stuff like that you always want to make sure you're using this try finally pattern and so the with statement is a way to abstract that away to factor out that functionality so that you don't have to write that every single time you open a file or you create a file now how does this actually work behind the scenes right because this seems like a really nice feature and in my opinion it's a highly useful feature in Python but how do we go from this to that behind the scenes and this is what I'm going to explain to you now so how can you support the width statement in your own objects because there's really nothing magical or special about the way open works there's not some magic sauce that's only available to Python built-in objects you can support the same functionality in your own programs or in your own objects by implementing so-called context managers so what's a context manager really what it boils down to is that it's a simple protocol or some interface or contract that your objects follow so that they can be used with the width statement and I'm going to show you a simple example here so this managed file class basically emulates what open did there and how we were able to use it with the with statement so there's two methods that an object needs to support in order to be used with the with statement and that is the enter in the exit method so it's a very very simple contract a very very simple interface and with this managed file class here we can go ahead and say ok with managed file as F and that looks exactly the same as the open function call looked earlier and now we can actually go ahead you know and write something to that file and we're going to get the same result the file is automatically going to be closed all right so I want to talk a little bit more about the steps that Python takes behind the scenes for for this example to actually work so you can see here we defined this managed file class and we've got this constructor here that just assigns the name it remembers the name for the file that we want to create and it doesn't actually open the file until the enter method is called and that's kind of how you want to structure your context managers right so that really the resource gets acquired when the enter method is called and it gets released when the exit method is called and then the exit method also takes additional parameters that will tell you about some exception that might have happened in case you want to you know inspect that log that or do something with that I'm not doing that here just checking like hey did we actually open anything if so then we're going to close the file and now obviously you know with this example you need to remember that this is sort of a useless wrapper around the open function here because the open function already pretty much does that as when it functions as a contacts manager right so this is merely an example to illustrate that but you could imagine this would be some other kind of resource like a database connection so now what's going on here in this example because when we create an instance of managed file we're not actually immediately calling the enter method so I could also do something like this you know I'm going to call this MF which is not an ideal name but for this example it'll work so I'm creating this managed file object and up until now we didn't actually call the enter method right you can see that here with this exception because the object doesn't have a file attribute yet we didn't assign that yet because we didn't call enter yet or python did call enter yet now when I go ahead and do with MF and then I can say something like with MF as the file then I'm going to enter this context where the resource is acquired and now I would have to go the file dot right hello dot txt talk about what's going on here right so we can see here with this statement with MF as the file behind the scenes this is going to call enter and then it's going to assign the return value so this is going to return the actual file it's going to assign that to this name here the file and then within the context off this manage file I can use the underscore file to actually write to the file right because this was the actual file object that I needed to write to the file and then when we leave the context it'll like right after I'm making this call here and we're going back one indentation level leaving this context then immediately Python is going to call the exit method or the dunder exit method all right so I hope that gives you a better idea of how these context managers actually work behind the scenes now I just explained to you how class-based context managers work but this isn't the only way to support the with statement in Python and this isn't this is not the only way to implement a context manager so there is the context Lib module in the standard library and it provides a couple of abstractions on top of the basic context manager protocol and this can make your life a little easier if you're trying to implement a use case that matches well with what the context Lib module offers so here's a quick example of what you can do with this so I'm going to re-implement the same managed file functionality using the context Lib library so there's a decorator in there called context manager and this thing is is highly useful so what that allows you to do is you can define objects that follow the context manager protocol that you can use with the with statement simply by writing a generator so you know when looking at this code again you can see our familiar try and finally pattern here so where I'm acquiring the resource and then I'm yielding it and then later I'm closing the resource and what happens is that this app context manager decorator will turn this generator function or this generator that I just defined here it will turn that into a full-blown context manager that I can use with the with state so again I can do something like this I can say with managed file hello dot txt I'm just going to call it f and and I can say I could write something to the file and of course we can also you know write write more stuff to the same file and now when I run this we're actually writing to the file if you're wondering what this is so this is just the number of characters written because write returns how many characters were written to the file right and it's just kind of leaking back into my interpreter session here so with this technique using contact slip and using the context manager decorator you can write some of these context managers a lot quicker I guess the downside is a dad for someone to understand this piece of code they would have to have some basic knowledge about decorators and they would also have some basic knowledge they would have to have some basic knowledge about how generators work in Python right so when you compare that with the class-based implementation you could actually say hey on the class based implementation communicates what's going on much more clearly for someone who's not an expert at Python so depending you know how how your team feels about this and what everyone's expertise level is it's kind of a fine balance with them what what concepts you want to pull into this you know personally I think this is pretty beautiful right this is awesome this is very pythonic but on the other hand it requires someone to understand many many more concepts in order to actually you know understand what's going on looking at this I mean they're going to still use it the same way so they might not care but I just wanted to mention that because in my mind code always needs to communicate what's going on or like code is communication and this is a great example where you can see where the trade offs are right and I can't make those decisions for you but it will be something that you need to decide within the context of your team the project you're working on and so on right duh just wanted to mention that all right so I hope that gave you a good overview of how these context managers work how the with statement works behind the scenes and what you can do with them there's a couple more really cool things you can do with context managers like you can write some really really nice ap eyes with that and I'm going deeper into the subject in my book that I'm working on called Python tricks a buffet of Awesome features in Python and if you're interested in the book the link is in the description so be sure to check it out and I'll talk to you soon thanks so much [Music]

Channel: Real Python

Views: 49,291

Rating: 4.9402514 out of 5

Keywords: dbader, pytricks, python, python trick, python programming, context manager, with statement, python tutorial, python with, python context manager, python with keyword, context managers, __enter__, python 3 with, python open as, python with open as, python with open close, __exit__, python __enter__, python enter, python __exit__, context manager protocol, __enter__ and __exit__, contextlib, design pattern, python design pattern, learn python, clean python

Id: iba-I4CrmyA

Length: 12min 39sec (759 seconds)

Published: Tue Feb 28 2017