Exception handling tips in Python ⚠ Write better Python code part 7

in this video i'm going to talk about exception handling in python and how it works i'm also going to show you two advanced mechanisms that help you improve the way your code handles errors if you're new here and you want to become a better software developer gain a deeper understanding of programming in general start now by subscribing and hitting the bell so you don't miss anything an error condition or error is traditionally viewed as a condition that occurs during runtime but can't be handled by the normal flow of execution of a program it's not a bug because it's not something that you can fix in the code for example a file that can be found some device that you're expecting to be connected is not internet connection is lost an authentication token is no longer valid etc most programming languages make a distinction between syntax errors that happen during compile interpret time and other errors happening during runtime in python that difference is less clear because parts of the program may be interpreted only when they're needed and handling errors while interpreting or errors while running the program are both done using exceptions this means that in python there's more or less no way around it if you want to properly handle errors in your python program you have to do it the python way and that means exceptions whether you like it or not exception handling is a pretty old technique it was first added to the lisp language in the 1960s nowadays exceptions are a common pattern in many languages and it's considered the standard way of dealing with errors it's not without criticism though and i'll talk about that later on but let's first look at a couple of examples of handling exceptions in python i thought it'd be nice to do an example about an api so i use the flask platform in this case to create a simple api for accessing blog posts here we have the flask application it's very basic there is a hello world which is just there for testing the flask basic route there is the blogs route which fetches blocks from a database and there's routes that fetches a specific block given some id each of these routes calls a function from another file that's responsible for actually retrieving the data from the database and then before sending it back to the client i'm calling the jsonify function which is a function from flask that allows me to turn the data that i get from the database into a json format so that you can retrieve it with an http request as a basis for the application i created a very simple sqlite database that contains just one table blogs that has a few things like an id which is the primary key there is a date that the blog was published there is a title there are some content and there's also an integer actually kind of a boolean in sql land that indicates whether the block is supposed to be public or not and then what it did was just insert a few rows of data into this sql database and you can see there's actually a file here application.db that contains the actual data in this case this is the file that contains the actual functions that retrieve the data from the database so there is a fetch blogs function that simply returns all the blogs from the database but only the public ones so that's what this sql query does when it retrieves the data it then converts that data to dictionary so that's actually what's happening here so that what you get is actually an array of values which is each of the columns in the query result but i want to convert it into a dictionary like so so i can send back the data as a json object which makes a lot more sense similarly fetchblog gets an id and then retrieves the block with that particular id from the database and when you look back at the actual api definition you see it simply calls these functions now obviously you could put the code that is inside these functions directly into the api routes and for very basic simple applications that's perfectly fine but as soon as your applications become a bit more complex you kind of want to separate these things in my company for example this in particular useful because you want to have both a graphql interface to our back end as well as a rest interface which our clients are using so what we did is that we separated the logic of what the backend should do from the actual routing and the interface to the client which is either http rest like request or a graphql system let's start up the server and see how it works and this is the hello world api route that we have here now i can also retrieve the blogs and now you see we get back an array with data about blocks in this case there's just a single block what else can do is retrieve a block with an id so let's retrieve the block with this id stacy we've created a very simple api that retweets data from a database the problem is i don't check anything so for example if i write here an id that doesn't exist i get an internal server error you want to avoid that your clients see these kind of errors because they're not very informative it would be much better to give a 404 or something like that so let's improve this and the way to do it is to think about error handling and exceptions let's take a look again at these database functions what we're doing is we're connecting to the database we're executing a query we're fetching data from that query turning it into a dictionary and then closing the database and returning the result again that's for fetching all the blocks fetching a particular block it's very similar also database connection execute a query we get the results converted to data that we want to have and then we return that data but here obviously things can go wrong for example there might be a problem with the connection maybe the file doesn't exist there might be a problem that we didn't find any blog here or there might be any other kind of issue that might occur sqlite has different kinds of exceptions that it can raise in fact python has a whole hierarchy of exceptions the main exception class is called exception and basically every exception should be a subclass of that main super class and lots of libraries and platforms provide their own set of exceptions so for example sql lite has an operational error exception that you could handle the thing is that it's okay to kind of handle these low level exceptions like sql lite operational error that's fine but on the api level you preferably don't want to be handling those very low level exceptions because then the api level needs to know that the database is actually using sql lite and maybe if we change it later we don't want to go through all the api routes to fix the error handling so that we use the new library's error handling so the way you could solve this is that in the fetch blog function you would handle the low level database errors and then you could potentially raise new errors that are more suitable to the level above as a first step let's add a few custom exception types that we're going to use and in this case what can go wrong it's possible that a blog cannot be found and it's also possible that we're trying to access a blog that's not public so let's add a not found error and a not authorized error and both of these classes are subclasses of the python exception class let's start with the fetch blog function so now we're not dealing with errors at all but what we have to do is put this into a try block so that we can handle the errors if they occur to keep this example simple let's say for now we're only handling sql light operation errors so that means we need to add an accept clause to the try block and if we want to do something with that arrow we should also give it a name such as e or whatever and then we can print it for example and instead of now pushing up this operational error of sql light to the layer above we can now raise a new error and let's say for simplicity's sake we're simply converting any operational error that we get into a not found error the problem here is that this operational error is likely to occur somewhere in this part where we're connecting to the database and we're trying to execute a query so we potentially open the database and then it went wrong in the query but then because of the exception we're not closing the database anymore so one thing you could do is copy paste this code to the accept part but python also has a finally clause that contains code that you should always execute regardless if there is an exception or not so now in all cases the database is closed what we need to do now is be a bit more precise about the way we handle errors so here we're fetching the result of the query and that is potentially none because it's possible we can't find the block so we need to handle that case here so if the result is none then we're going to raise an error and this avoids the problem where we're calling this function to convert the result to dictionary with a result that is not the second thing we should do is check if we're actually authorized to view this block and that we can do after we've converted the result to a dictionary if the blog is not public we should raise a not authorized error and now we can go level up and actually handle those errors and send the correct http response that belongs with them so instead of directly returning the json data that we got by fetching the block from the database we're also going to check for errors here but here we're going to handle the high level errors which is not found error and not authorized error which i will also need to import so in case a blog is not found we can use the abort function to send a 404 response back using flask and similar for not authorized error we can send back a 403 and you may want to add more information to these particular messages that you send back if you want to let me switch to my browser again and let's reload this page so you see we get a not found because this block doesn't exist this was the block we had in database and there is another one a private block that we're not supposed to access and then you see we get a 403 back so here you see how you can use exceptions on multiple levels to make sure the right information is transferred often exceptions are being seen as a kind of hidden control flow which they are and that's pretty bad it's important to make sure that you're handling exceptions at the right layer and not pushing them all the way to the top where you don't know what that low level thing is we can do something similar for fetching all the blocks but then we can actually make it even simpler let's say we wrap all the code into a try block again like this and then we're going to do exactly the same where closing the database is part of the finally clause and this actually belongs there there you go and now what we can do is simply say okay we're going to catch any exception that might occur here connection problems query problems whatever let's print it out so that we know what it is and then we can send back just the empty array so if there is any problem here you will always at least get some kind of result potentially this is not exactly what you want maybe you want to have more precise control here so then you could add different kinds of exception handling or perhaps you want to communicate back to a user that there is a problem with the server but in this example because we're dealing with the exception on this level actually here on the error handling part we don't need to do anything because fetch blogs is always going to return some kind of valid result if you don't like this finally part of exception handling then python actually has another option for you and that's using context managers context managers allow you to have a bit more control over what happens when objects are created and when they are destroyed and you can use that to free up resources at the end which is very useful in the case of database connections python already has a few of these context managers for example for opening a file if you open a file and an exception occurs then python makes sure that the file is actually closed if you use the with keyword so we can use a similar procedure for opening and closing the database in order to do that we need to create a context manager object because sqlite does not have this as a built-in feature and what you need to do then is create a class that has two methods enter and leave and in the enter method you specify what should happen at creation time and the leaf method should specify what happens at destruction time so let's create a class called sqlite where we're going to handle this in order to work with the database we're going to need a file name so that's what we need to pass in the initializer and then we can add an enter and exit method in the enter method we're going to set up the connection and in the exit method we're going to close the connection and now we can use it with the with keyword let's see how that works for example in fetch blogs i could wrap this part into a width block and then i don't need to do this connection part anymore because that's what the context manager is going to handle for me now obviously we're still going to need a cursor so what you could do is that the enter method actually returns a cursor object and what we need to do here in the width part is that we need to add a variable so now we have the cursor that we can use here inside this width block and because the context manager is going to handle properly closing the database connection if something happens we don't need this finally parts anymore now personally i don't mind using the finally part in an error handling system but what i also appreciate about context managers is that you have the creation and destruction of resources in a single place and that just makes it a bit easier to organize everything in your code so let's start the server again and see if it still works and now let's fetch all the blogs and we get an empty list so actually it didn't work because i forgot to add a few parameters to the exit function so the exit function should have a type a value and a traceback parameter we're not using it here but it's still needed for it in order to work correctly let's try it again yes this is exactly what we were looking for before we finish this example i'd like to show you two more advanced things you can do with exceptions the first one is a so-called retry decorator you can imagine when you're dealing with database connections or network connections in general that sometimes it's useful to simply retry something if something doesn't work the first time it doesn't mean that it's broken it's possible that some packet got lost in a network or that there was some other connection error and perhaps you should just try it again and then maybe it will work and this is a nice function that i found that actually models this behavior for you so it creates a decorator that you can wrap around a function that keeps track of a number of tries puts a delay between them and if there is an exception it waits a bit then tries it again for a number of times and the way you can use it is like this so let's say you have a function that's going to fail because the only thing it does is raise an exception but you can put a retry decorator on top of it where you provide the kind of exception that you should handle inside the retry decorator and then also how often you should retry this so when we run this particular example you see that it's going to print out this message where it will retry again and again after the exception is raised and the way this particular decorator works is that it multiplies the wait time every time an exception occurs and then after four times it simply gives up so this is a nice way to add a more advanced exception handling behavior to your code i put this in the git repository by the way so if you want to use it you can simply find it there another thing you could do with decorators and exceptions is that you can automatically log exceptions so here you see we have a decorator that wraps also around the function where in case of an exception it's going to create a log and then it's going to log that to a file or whatever you want and it's going to raise that exception again so you can then properly handle it and the way you use it is that you have this function in this case this is going to raise an exception because we're dividing by zero but i put an exception decorator above it so it's also going to log this so if i run this let's see what happens so we indeed got this zero division error but it's also created a log file and that's over here and there you see we have the trace etc etc so this is a way to easily add log files to your application if things go wrong and you don't have to write a lot of boilerplate code for it so we have these exceptions problem solved right unfortunately exceptions are certainly not perfect the first problem with exceptions is that they introduce a second hidden control flow in your program and this means it's harder to grasp for programs what happens in the case of an error and it might break other things that you're not aware of and that's the second problem if you're not careful exceptions may result in resource leaks in case of an error not closing a database connection for example or not releasing memory that's in use a third problem is that exceptions potentially introduce extra coupling because high level systems may need to know about low level exception objects if you don't set it up right it's also not always clear what the error values are you have exception objects but those may contain different data depending on the type of error and you need to document that and finally you're never quite sure that your code handles all exceptions properly and this is very hard to test because exceptions can basically come from anywhere there are a few other ways of dealing with errors but they each have their own set of problems first thing you could do is just terminate the program too bad for the users you shouldn't have pressed that button you shouldn't have opened that file let's not even talk about that solution a second thing you could do is that if you have a function that potentially raised an exception you could return an error code instead of a value but that breaks a lot of type systems a third thing you could do is call an error handler function on error that's pretty common in javascript but you quickly end up in callback hell so it's not so great either then what you could also do is return a dummy but legal value from a function but set a global error somewhere else like a global error stack but the result of that is potentially a lot of coupling in your code because every part of your application needs to access that error stack final thing you could do is return a legal value but also set on error flag as a function result that's called deferred error handling it's how the go language works that's probably a close contender for least bad error handling solution next to exceptions it does make your code look slightly more verbose and you're actually forced to address any errors because they are functional results and that helps with stability but it doesn't always lead to more readable code and there's less room for just a quick and dirty script but perhaps that's not a bad thing i hope you enjoyed this video let us know in the comments below what your tips are for handling errors also check out the discord server for more in-depth discussions about software design and development thanks for watching take care and see you next time you

Channel: ArjanCodes

Views: 91,876

Keywords: exception handling tips in python, python exceptions, python tutorial, python programming, computer science, learn python, how to write clean code, clean code, exception handling, exception handling in python, software engineer without degree, computer science crash course, python exceptions explained, python exceptions for control flow, python exceptions vs error codes, python exceptions tutorial, python exceptions custom, errors and exceptions in python

Id: ZsvftkbbrR0

Length: 21min 46sec (1306 seconds)

Published: Fri Mar 26 2021

Reddit Comments

[u/_szs]

Interesting topic, well explained. One question, though:

Is it a good idea to put a with-block inside a try block? Will the exception (e.g. failed db connection or missing file) even leave the context?

[u/war_against_myself]

In one example where there’s a database connection ‘conn’ if you do finally: conn.close() you might get NameError. Can happen in any try except block where exception is raised before a variable is assigned. I usually create my variables beforehand as ‘None’ to help avoid this. Then if conn: conn.close()

[u/BoppreH]

Mostly good video, but has severe problems:

On different error handling strategies:

• 19:54 terminating the program is a perfectly fine solution, not a punchline. From the documentation of a language known for its extreme reliability: "When a run-time error or generated error occurs in Erlang, execution for the process that evaluated the erroneous expression is stopped." The trick is having robust message passing and independent actors.
• 20:13 Returning an error code breaks a lot of type systems? Even C has union types! No idea what is meant here.
• 20:44 I've never heard the term "deferred error handling" before. Are you perhaps confusing it with another Go feature, the defer statement? It's used for managing resources, sometimes in the face of early function returns (e.g. errors), but that's orthogonal and it's definitely not comparable to the other methods presented. Also, Go errors are almost always values (newly created objects with context), not flags (numbers or enums) as mentioned.
• ?? No mention of Algebraic Data Types, arguably the gold standard for error handling?

And two big coding mistakes:

• Barely any mention on why except Exception: return [] is a terrible idea, and why it's not accepted in most code reviews. Do you really want to return an empty list and continue execution when you have typos in your function? This is the exact type of mistake that I expected to see addressed in a video on "Advanced Exception Handling", not shown as a normal thing.
• A glaring SQL injection that was shown multiple times, with no disclaimer. This is too dangerous to show even in a toy example, and the safe version is actually the same length, so no excuses there.

# Script kiddies will delete your database:
cur.execute(f"SELECT * FROM blogs WHERE id='{id}'")

# Perfectly safe:
cur.execute("SELECT * FROM blogs WHERE id=?", [id])

I find error handling strategies in programming languages fascinating, and this video does cover a lot of it, with eagerness and good faith. But I think it unfortunately has far too many mistakes and omissions.

[u/DehiXeM]

The retry() decorator was a new one for me,

I liked the context manager approach aswell,

​

Thank s for the video

[u/milki_]

I particular liked the @retry decorator. Been looking for such an implementation. (And this should really go into the wiki.python.org somewhere. Since you know, commonly forgotten in youtube descriptions. wink,wink)

This opens up a bigger question though. Why aren't RetryableExceptions a thing? Or even some standard on translating exceptions to user errors / docs / links or whatever.

[u/jringstad]

ADTs are IMO far better than the error handling mechanisms you discuss.

• They don't bubble down the stack like NULLs/Nones

• They don't bubble up the stack like exceptions (or call remote error handlers, which fundamentally amounts to the same thing -- someone elsewhere is supposed to handle an error they don't understand the context of)

• They don't require state of any kind

The only thing is that you really want language level support for them to make them pleasant, similar to how rust does it. Allow pattern matching on them, and easily convert them to exceptions.

[u/Wuncemoor]

Why must everything be a video