Python Debugging (PyCharm + VS Code)

hello and welcome to M coding I'm James Murphy today we're talking about debugging there's no getting around it no matter how smart you are no matter how experienced you are you're still a human and you're going to write bugs and whether you wrote the bug or your teammate did inevitably it's still going to be on you to fix it and hey if you're happy just throwing in a print statement and that helps you solve your problem that's totally fine this is called print debugging and it never goes away but a lot of the time print debugging is is going to be very frustrating and using something like a live debugger like the one that I have open here is going to be a lot easier at a fundamental level all they do is allow you to step through and pause your program at particular points so you can inspect the state and see what's going wrong even basic live debugging is extremely useful and not that hard to do either but if you've never heard about it then you might not know it's even possible so today we're going to take a look at sort of the main features of a live debugger and use a live debugger in order to debug this grocery list application and don't worry I'm going to show you how to do it in VSS code or py Charm unfortunately if you're using Vim or emac you clearly enjoy the pain so you're going to have to figure it out on your own and as you could probably tell from watching most of my videos I prefer pie charm so if you'd like to win a professional pie charm license go ahead and comment down below thanks to Jet brains for providing the licenses but don't worry the debugger works just the same even if you're on the free edition okay so what does this grocery application do basically I have a list of recipes so here I have a spaghetti bologan and a list of ingredients and quantities for those ingredients yes I realize that a list of ingredients is not the same thing as a recipe sorry I hope it doesn't make the video totally unwatchable each of the ingredients listed in the recipe is also listed in the ingredients list in a separate file and in the ingredients list it says Where in the grocery store that ingredient would be found so if I was looking for an avocado I should look in the produce section when I run the application I get a list of the recipes and it asks me what I'm shopping for so let's say I want to make uh two spaghetti a Caesar salad and a veggie stir fry I just put that in and hit enter the printout then contains the ingredients for all the recipes that you're trying to make so that you can go to the grocery store and just buy all the things they're also grouped by which section in the grocery store so that when you're out shopping you can just get all the things by section because they're physically closer to each other our beta testers have reported up to a 50% reduction in time spent in the grocery store but that was the already fixed version just for demo purposes what we're going to be dealing with is a very similar but slightly broken version the main difference between this one and the previous one is that this one doesn't work like a lot of code you're going to spend time debugging you didn't write this code and you don't fully understand it and quite frankly you don't have time to read all of it in order to fully understand it you'd like to be able to just debug the minimal amount that you need to in order to understand what you need to understand in order to fix the problem we'll get there as soon as possible but first let's just see how to use the debugger at all unfortunately this first step depends on which version of python you're using if you're using 311 or less then you don't need to do anything extra but in 312 they changed how debugging works and py charm doesn't really support the new way that well so go up here into help or use the keyboard shortcut of course uh go to registry and find this python. debug low impact monitoring API that is set on by default if you're in 312 go ahead and turn it off for me it was causing spous crashes but hopefully in 5 years when your company starts actually using python 312 this issue will be resolved okay let's begin the first thing that you'll notice about the debugger if you try to start it by clicking on this debug icon is that it doesn't appear to do anything different than just running the program I still get the same choices I can still pick my menu items and I still get the same error and how does this help me in order to do something useful with a debugger you need to set break points these are places where if python reaches this point in the code then it will stop and show you information the simplest thing you can do is set a breakpoint at the very first line in your main function now when we click debug python reaches that line and then stops basically at the very beginning of our program the main thing we can do at this point is to use these controls down here so we have rerun stop resume pause step over step into step into my code step out View breakpoints and mute breakpoints now normally you would memorize the keyboard shortcuts for these things pretty quickly but for the sake of the video and you being able to see what I'm doing I will do my best in order to actually click the buttons step over is basically what lets you run one line of code so right now we're paused at line 80 and line 80 has not run yet when I hit step over then it runs that line and then stops at the next line the previous line defined a variable and that variable now you can see down here in this list of basically all the local variables we can go in and inspect it we can see that there's nine items in the list and you can just dig down further and further and see different properties this is a great way to get a feel for what's happened inside the local scope of a function step into is similar to step over in that it kind of runs one line of code except if the line of code involves some sub things then it will actually go down into that sub thing so in this case the next line is this load ingredient locations so when I click step into we actually go into that function and normally you could continue stepping into further and further down functions but in this case the next thing that's going to happen is it's going to basically call this open function but this open function is not written in Python this is a python built-in that's actually written in C so if we actually try to step into that you'll see that our live debugger like pointer to where we are kind of just disappears now the program is still running it's still debugging and if we keep going we'll eventually get back to python land you can see now it's came back but it didn't show us that inside of the open function this is just something you get used to you typically don't have any need to be debugging very lowl functions anyway let's just step over that and you can see we're now on this Json load line and that actually is written in Python so we can go ahead and step into that and sort of see the implementation of um this thing that's inside the Json library and we could go down sort of as as far as we could possibly try and eventually we'll get to a point where we can't go down any further we can step through this line by line if we really want to but again this is basically lowl code that we didn't write and we probably don't need to be debugging it unless perhaps we're just interested in how does the Json Library work so if you've ever stepped too far down and just want to go up a few levels in the call stack you can see the call stack right here by the way where I can see all the way up from uh the start of the module into main into my load ingredients function then into the J on Library into the load function all that uh you can you can see here and you can see all the variables all along the way if you're ever too far down and you want to just go back up you can hit uh the step out button that's what that's for so let's step out of that out of that and now we're back into R code let's just step out of that one more time so doing this you can basically step through your entire program and it still works as normal you can give it input and you can just go line by line until you eventually find the error or the crash you can see in this case I tried to step over but instead of sent me up to main what's actually happening here is this is what it looks like when you hit an exception this is an exception being raised and eventually it escapes outside Main and we get the error that we saw if we ran the program normally if we wanted to start over we just hit debug again and it puts us back at Main but going step by step through this entire program is probably not the best way to find this bug especially since we have a trace back which is presumably close to the origin of the area let's just take a look at that error one more time we gave it some input and we get a type error at this particular line so let's go there the trace back is telling us that we're inside this function and that the error is originating here that whatever this evaluates to is not inside of this list or this map so what we'll do is just set a breakpoint there and click debug we'll go ahead and just execute the program normally until we get to that point in the program and then now we see that we stop right before that line the one that was causing the error in most debuggers you can hover over stuff to see so like this is a dictionary with key name that's ground beef and quantity one pound I can see that same information here inside the local variables of this function and if I really want to dive deep and actually evaluate expressions in real time I can just type them in here inside the debug console so if I want to look at this variable then I can just print it out like that or if I want to look at what is the name evaluating to then I can literally just type in whatever expression I want and it's like I'm at an interactive session but at this state paused okay the ingredient name is ground beef so let's try to look that up inside of this map and we see well there's the error reproduced we get a type error this is a list but we're trying to use it like a map and we can confirm that here uh the ingredients it says the type here is a list and we can see it has 10 items if we we go ahead and look at our ingredients data we see that indeed our ingredients data is just stored in a big list where each element is a dictionary of the name and where the ingredient is but from the way that it's being used in the code it seems like what we really want isn't a list of all the ingredients but rather a map of every ingredient to where in the store it is alternatively for each ingredient we could do a linear search trying to find that ingredient somewhere in the list but in this case a map makes more sense so then we need to go on a little mini mission to find where this was actually set where did this data come from in this case it's no surprise that the all ingredients locations is the one from the main and that was just from this load ingredient locations if we jump there then we see that indeed we're just loading it as Json meaning we're getting the list form uh instead of the map that we're looking for now again the kind of fix that you do is up to you do we want to change that one function do we want to change this where the data is loaded so that it's used a different way everywhere in the application or do we want to change the data itself totally up to you in this case I'm going to go ahead and say that the place that we want to change it is in this function where we load it so we apply our little fix convert the list into a dictionary and then let's go ahead and check did this solve the problem let's say what do I want to eat how about uh tacos uh so we didn't get the same error we got a different error and this is an important lesson to learn even if you fix something sometimes you're not done sometimes there's more than one thing wrong in this case once again it's on that same line but it's a different error here we're getting a key error like salsa isn't a known ingredient let's keep our breakpoint and just debug it again this time though when we get here and it pauses we're actually not right before the error occurs remember it was happening with salsa but right now we're with ground beef I'm actually inside of a loop right now and as you can see it's sort of just going around the loop so we're at lettuce now we're at Tomatoes now we're at cheddar cheese and so on and so forth I'm sort of having to step my way through in order to get to salsa and actually oops I went one too many and now I passed the error this is a perfect situation for conditional break points I want to stop at this line but not every time I get to this line I just just want to stop here if the ingredient is salsa that's a perfect use case for conditional breakpoints so go to our breakpoint and right click and we get this extra menu and it has this condition in it here it will only stop at the breako if the condition is true so in this case I'll take this ingredient name and that needs to be equal to salsa then click done and now let's debug and this time when I hit the break point I skipped all the previous ones and I I do indeed stop at Salsa Salsa was giving us a key error so let's look inside our ingredients and see is salsa actually there so it's inside the all ingredient locations and we do see a Salsa Salsa is right here but it looks like the issue is that the name here is lowercase salsa but for some reason it appears as uppercase salsa within the ingredients list based on the rest of them it seems pretty clear that they were probably all supposed to just be lowercase I mean at the very least the way that it's referenced inside the recipe probably just needs to match whatever is in the ingredients list but since everything is lowercase except for that one ingredient I think what we really want is to just lowercase the salsa swag this is literally not planned so important lesson number two the problem in your code isn't necessarily in your code it could be in the data it could be from user inputs it could be environmental variables it could be some kind of assumption about the operating system or file system be prepared to check all of your sources the fix for this is twofold number one of course fix the data it should be lowercase salsa but number two validate the data so let's go back here and add in a verify that ingredients exist function it just Loop through all the recipes and looks at every ingredient in every recipe and make sure that that ingredient is found in the ingredients list then we just need to add that check into our main function so that if this ever happens again we'll know immediately what the problem is it'll just tell us hey this ingredient wasn't found even though you're still going to be getting an error it's a lot better this way the earlier in your program that the error occurs the easier it is to find and the easier it is to fix let's debug the program one more time and get to that line with salsa and we get here and let's just step over one more time and we see we don't crash when we get to salsa this time so that's good in cases where you have multiple break points and you want to jump between them or especially when you want a quick way to jump out of a loop without jumping out of the entire function here's something you can do you can set another breakpoint at the place that you want to go to and then hit this resume or sometimes it's called continue what this does is just keep running the program until you hit another break point since there was only one salsa in the list this skipped through all the rest of that Loop and allowed me to continue debugging from their on I can now step over and just keep going from there but since there's not any other break point that I wanted to stop at let's just resume and let it finish so we can see that in fact it did finish but it didn't quite do the right thing it looks like every ingredient got a section to itself even the ones that were in the same location like avocado and cilantro are both produce they should have been grouped together also the print out of the sections looks a little bit weird too and this is another important lesson that we need to learn sometimes things go wrong but in a way that's not apparent through an exception so this one will be slightly more difficult to find but still not too bad all right for this last one let's switch over to vs code so you can see how to do it here too in vs code you're going to want to go to this run and debug and then say create launch. Json file we'll use the python debugger and select this first option which is just a plain old python file as you can see it created this debug configuration with some default options and it put that in this launch. Json file inside your vs code directory for some of you this might work but as you can see for me this is actually going to cause a problem if I go to run and debug and I hit the start debugging you see I actually get an exception right away basically no file or directory recipes. Json it can't find my recipes but this is the same directory structure as I had before so what's going on the problem which you can see from the command that it ran here is that it actually ran the command from the root of my project directory not from where the file was I do think it's a little weird to have the defaults be when you say run the current file to not have that file's directory as the current working directory but whatever a default is a default we didn't have to do this with py charm because py charm makes the current working directory to be the directory of the file by default uh but for this case we're just going to have to set it manually so we just set the current working directory to this special variable file durame which is the directory containing the file that you just ran I also typically want this just my code variable to be false this is another one that I don't really understand why the default is true if this is true then vs code doesn't let you step into or pause on a function that you didn't write I think diving down into the libraries that you use not just the ones that you wrote is a great way to learn how those libraries work oh yeah and those libraries might also have bugs in them that might be the source of your bug in any case let's see if we can get the debugger to work this time and it does appear like it's working and we are able to successfully reproduce the error that we had before at this point we don't have a whole lot to go off of but at the very least we can see bad behavior when it's printed so we're going to go ahead and start at this function which looks like it's the only place where it's actually printing something let's just set a breakpoint there and see how it goes the boxes are in different places but it's all the same stuff we have our local variables here which we can dive into in the same way we can click through and see all of our variables uh watches these are just Expressions that you can watch it's kind of like an expression that keeps getting evaluated every time you pause so you can see a certain value uh I don't typically use these very much we have our call stack that we can jump in and out of and then our break points that are current currently set all of our controls it's the same stuff we have continue step over step in step out restart and stop so let's go ahead and step into this print grocery items by location we Define a key function we Define another key function then we sort our items by basically the where and then the name and then we group them by the where then the name uh and then we go through and we print those out so not a whole lot going on in this function this is a great point to let my ID help me and notice that this where key is gray out meaning this is an unused variable I defined a function but I never used it another thing that I notice is that this variable is just called where here which kind of lines up with what I was expecting I wanted to see where each thing was each header was supposed to be a location like produce not a tupal of produce and an ingredient let's just step through to get to the first wear and indeed we see this Tuple containing two items the wear and the actual ingredient instead of just the wear if we wanted to play around and investigate further this debug console is the similar thing to what we had in P charm so for instance if I printed out wear here we would see that tle now we can start to put things together the looping looks correct and it looks like I meant to use we but I'm not using it which means the groups should have been grouped by where and where is an unused key so I'm guessing that this group by instead of using where then name should have just used where which was an unused variable which would fix both of the two weird things that we found I can kind of see how this happened because it's very common to First sort by a key and then immediately Group by it because Group by works on adjacent elements so if my keys were like a a a a then BBB B then a a a a I get three groups a then B then a again it doesn't do the Sorting on its own so if you don't want any duplicate groups then you need to sort first hence why it's a common idiom to sort by and then Group by but in this case I actually want to group by something different than what I want to sort by I want to sort using this key so that they're sorted first by their location and then within each location they'll be sorted alphabetically but when I group them I just want to group them by location not by individual ingredients okay so let's see if that fixed it we changed the code but even if you change code after it's already started running those code updates don't come into effect so we do need to restart here go ahead and try again and this time we'll just go ahead and continue and it looks like everything printed out correctly this time all the produce are grouped together so there you have it that's the basics of debugging in vs code and py charm but I do have one more piece of advice for you in how to make your debugging more effective this was a pretty small project start to finish it was just over 100 lines and because it was so small we didn't have a lot of friction in just running the program reproducing a bad result and then fixing it but in the worst case potentially anytime you make any change anywhere in the project anything else anywhere else in the project could now be broken of course it's not usually that severe it's not like every commit you break something I mean hopefully not and the larger the code base is the more developers working on it the longer it's been since you last looked at that line of code the less practical this just run the debugger and hope to stumble upon the bug approach is that's why especially in a professional setting you want to reduce the problem down to a much smaller Problem by writing tests if you have a rock solid Suite of tests then you don't need to debug your entire application you just need to debug the load function or the load ingredients function or the choose recipes function you reduce the scope of debugging everything down to just debug this one function and even if you can't get it down to debugging literally just a single function it's still a much smaller surface that you have to cover every single one of the bugs that we debugged in this video could have been caught by a simple test and that would make it even easier to debug because the best time to debug is right after you wrote the code that caused the test to fail so if you're already going down that route of being a professional software developer or if you're just getting into larger projects I definitely recommend spending the time to learn how to do testing check out my automated testing video it's a great place to get started anyway don't forget about the giveaway if you like a chance to win a professional license to pie charm or one of jet Brand's other professional idees then drop a comment below #m coding for a chance to win I'm James Murphy and my company is M coding we also do software Consulting Contracting and training so if you're in need of those things please do Reach Out visit M coding. also did you know this channel is completely funded by patreon and other donations by viewers like you so thank you to everyone who's contributing if you'd like to support the channel visit patreon.com/crashcourse

Channel: mCoding

Views: 32,595

Keywords:

Id: COa-JHYuW3M

Length: 24min 17sec (1457 seconds)

Published: Wed Feb 14 2024