Build A Forecasting Page - Building SaaS with Python and Django #95

hi welcome to building sas with python and django my name is matt lehmann and on this stream we build django apps we're going to continue on my homeschooling application that i'm building and continue on some feature work that we were doing last week so if that's interesting to you stick around if you're catching this on youtube please like subscribe comment do all the things that make youtube happy there's links below in the description that you can check out that might be helpful to you where um i just generally put stuff that's useful i think and um there's plenty of ways to keep in touch with me i've got a website above listed at mattlayman.com um mailing lists the whole bit all that i do want to thank my patrons um you can join me join them at patreon.com and thanks to my patrons of rupert andrew abdulaziz patrick eric and philip i appreciate your support that's really awesome um so we're going to continue with the the feature that i was working on last week we were doing some forecasting page for the the application um the as a reminder if you knew uh this application that i'm building is a homeschooling app so it's a support of my family my both my kids are homeschooled and my wife is the user of this application and it builds a schedule so you feed it tasks and it puts stuff on the schedule of these are the dates that matter for your school year but the challenge with that approach is since everything is not set to specific dates you don't exactly know when stuff is going to end so you need some kind of ability to see do i have enough stuff to fill up my school year and or am i going too far like maybe i'm i'm falling behind and all the tasks that i put in extend beyond and into the next school year and that's not good either so the objective of this forecast page that we're building is to provide that at a glance view of here are all your students here's when they are expected to finish all of the coursework for the school year and give that feedback to the user so that they can know all right maybe i need to add more stuff for this course or i need to figure out how to condense my material here because they don't necessarily know this was a requested feature by my spouse who is the primary customer that i have and last week we were working on refactoring some of this logic was actually embedded in a different page kind of projecting out one course page at a time of here's all the dates that the student has anticipated to finish a task but we need a synopsis kind of view that will roll together all of the courses that a student is in for each student in the school and give them the answers of you know here is the date that this is expected to end that's the goal of this page that we're going to work through and we've got some code to do that but we have to finish up the wiring of all of that stuff so that's the plan if you have any questions feel free to ask me anything about python django or web development that's what i'm here for the chat is open i do ask that people be polite and civil i will ban you if you can't be that way most people are not that way most of the time we have a really good conversation so feel free to use the chat to ask questions and let's dive right into it shall we um so this particular task i believe is running is in this milestone i don't remember which task number it is it's this end date forecasting report is what we've got here and there's not a lot of details here so we have to figure out what's the what's going to be on this page where it's going to go and first of all get ourselves right back into the right view i think i'm in the wrong file so we'll start by getting ourselves on the view and here it is right now it's just a template view there's nothing in here that's too fancy aside from the actual school year object that populates the view and we can fire up the local web development server and see what's going on so let's do that to orient ourselves to the material that we need to see and we'll get going from there so i'm going to log in with my test account which is just a local account that i have on this little database that i have on my machine which has some data kind of preceded in here and we're going to jump to the school year page and here's the school year and so the new part and and one of the pieces that i was wrestling with last time uh we had this chat was um how does how's the user going to get to this and i'm not super happy with the ui right now and we talk through well if i add another button it's going to look kind of clunky and where did where is the where's the navigation going i don't really have good patterns kind of set for some of these detailed pages that are meant to be just a bunch of information but for the interim what i did is just put this link right here which while not the best is a reasonable spot to put it so we're going to click on this page and you can see that what it is intended to be is a report of the entire school year and we're calling it the forecast report so this page as i'm imagining it will in its first incarnation be pretty simple it will be a student's name followed by all of the courses they are in for a school year um for their grade level i guess and then next to that will be at a date that says you know this is the expected date we could get fancier and start highlighting the colors of these dates depending on when the school year is slated to end or something to that effect but that's that's kind of a phase two of this process the important part right now is getting the data in there in fact i think my my wife is actually pretty eager to see this feature be complete and be out there and so i i don't think she's going to be super picky about the the format granted we don't want to look ugly but we don't have to fret a ton about the actual feel that can come later as you might want different or more information in this report so i think the objective tonight is to get the data and then feed it to the right tool and display it and and maybe step one even before then before we do the calculated data which is the date stuff is just to display the courses for the actual student and so we can look at the previous views to help guide that and figure out which things we need to call i think it's um yeah i think it's a a method that exists on the student or somewhere we're gonna have to find it the there's some challenges here with the way some of the data is modeled uh under the under the covers i can bring up we we have a diagram that we often like to look at the stream that is the data model for this system this is generated from a tool uh so i did not build this myself which is why it's you know it's kind of all over the place and it changes every time i generate it if i change the any of the modeling so we'll do our best to look at it but what we need to do is we need to get these courses courses are tied to a student via the enrollment where is it so here's a student over here here's this enrollment model and here's a grade level the grade level a student is in a grade level for the year and a grade level is a container for a set of courses and that relationship gets a little complicated because that's actually a many-to-many relationship because a course can be in multiple grade levels so i've got two kids for example they both take an art class and there's there's not any differences between the art that my wife is teaching them at the moment because my kids are pretty close in age so they have a shared art course so the modeling is a little um funky i'll say to get the actual data that we need so there is a method that i have somewhere to get the ordered list of courses and that ordering is uh is also something that she wanted to be able to control in the display so all of this stuff kind of adds together and and adds some complexity to what we need to pull uh but with any luck we'll um be able to get this kind of hooked into the context data write some tests for it prove that it will display for each of the students in the school year and go from that point all right so where are we going to get this stuff from and what's the good starting point the as i said in the model diagram the the key entry into a it's connecting a student to the school stuff is through this enrollment model so we have to query on the enrollment and we have to tie it back to the school year so that's the piece of data that we're starting with that we actually do have in this view and it you can see right now it's being assigned directly to the context that's probably not really what we want as a in terms of being a local variable so we're going to probably first maybe start by refactoring this to say let's take this and drop that there and just say this is going to be school year this is going to be the local variable that we'll create and we will change this part to be school year so that's step number one and did i okay the reason to do that is because accessing the context every time we want to access the school year while doable is kind of inconvenient so we're going to use this as our handle to make further queries the other thing i want to do is get the well think about the data before we actually get too far so when this gets displayed it needs to display each student so we need to be a loop over the students and then from the students we need to be able to loop over the courses and as we're looping over those courses we have dates associated with that so we the the most basic flow that i'm thinking of is is get the students which comes from getting the enrollments for that school year and then from each enrollment we can pull out the student sorry excuse me and then we can call the courses together as well and then merge that together with the actual tool that does the calculation on the course all right that's that's the rough idea i have in my head if that didn't make any sense to you don't worry about it i i understand that i've got more modeling context for this application than than anyone really because i'm the one who sits in it all the time and i designed it so let's bring up the schools test views file return to that and we should already have we've got this test case already here that has the get request and the other thing i guess we should check is we want to insert so i like to add some assertions to my test case to prove that the since i'm not necessarily doing full template tests of looking for content and data for looking for content in the rendered output i should say sorry um i like to have some asserts in my get request and such to prove that i am following a data contract so it kind of tells future me that hey here's the context that the template actually cares about and by having a list there i can prove to myself or or at least have some confidence that that they that the template is going to be using that piece of data so let's start by adding the assert and we're going to add that there's a in the context there is a school year and it's combined uh in that no underscore way because this is using um i'm kind of recycling from a previous template that is the detail view and django when you have a multi-word model instead of like putting them together with an underscore kind of kind of jams them together so school year is no underscore there which is why you might think like why is he doing that for the context it's because of those detail templates so we'll run that test and prove that that part is in the context now we want to have this structure and in my mind the structure is a list so it's it's probably let's like write out what i think the data is going to look like kind of scratch through it so it's going to be a list and in that list will be some dictionaries and the dictionary will say one key will be the student itself and that'll be student and then the other key will be the courses um as i'm talking through that yeah yeah we'll we'll do that for now and the courses will also be a list but it won't just be a list of course objects it will be another dictionary i think and it will be what the first key will be a course and the second key is going to be the expected end date expected completion date she was very clear to me that that it shouldn't be called an end date because this content can change and so yeah i didn't want to convey that um so we'll say expected completion date and then have some kind of date and so that's the structure that we're looking for i'm i'm not formatting this properly it seems but yeah none of that data truly exists yet and you know this is just to give you an idea of this is the data structure that i expect to render in the template and and this is the responsibility the obligation of the view to put that data structure together okay so first to get closer to that and what should we call that um i think the overall data structure list maybe just called students even though it's going to be students plus all the info about the students but it's a good key to iterate over so let let's write this test and assert on what we expect to see so we want to assert that the context has uh students and we want that to be equal to to start let's do a dictionary an empty dictionary because we don't have any students created in the test yet but this will immediately fail because the student's key isn't going to be in there so we can do the first thing to make the test passed the simple thing and that's to say students is equal to an empty list i mean this is like pure test driven development here where we're doing the simplest thing to make the test pass i don't i'm i'm not sure why i'm choosing this approach tonight but i am i kind of vary depending on how i'm feeling and this time it's nice to kind of piece this together since i'm not really 100 confident on the data structure so that now passes but what we really want is something different we want to have a um a student that's based on an enrollment and so the next piece to do is to go into our assert and we're going to change this now to say this is going to be an uh dictionary and half of it's going to be the student we're not going to get to the courses yet we're just like one piece at a time and this doesn't work yet because student that doesn't mean anything um in fact what it's really going to be is we're going to use an enrollment factory so we'll do access it this way and it'll be an enrollment student so the enrollment doesn't exist so we need to put that together we need to say enrollment is equal to the enrollment factory and that factory is going to take the grade level um school here and pass in the school year from the step above and that's our way to confirm that we're pulling from a grade level that we're allowed to access so i'm doing that because knowing in the view like you can see there's this get object or 404 step here that is is pulling based on an access control so we're already making sure that we're gating this so that other users cannot just access somebody else's forecast if they wanted to that would be impossible to do because of this check to make sure that the user who is logged in is the admin of the school so by doing the wiring of the data this way we've got proof that there's an enrollment there so now i'm convinced that's the test structure that we want to start and we can say okay give me the test and see if it passes it doesn't because we haven't done it we've got a hard coded list down here so the next step that we need to do is we need to get our students and so the way i'm going to do that is i'm going to say enrollment or enrollments is really what we're looking for and we want to pull from enrollment objects filter and we want to filter on the grade level school year and that's going to be equal to the school year object that we already fetched all right so the other thing that i want to do to make sure that we're not because we're about to do is pull out the students out of the list of enrollments which means if we were to do any kind of list iterating stuff on it if i went with this implementation right now let's say a family has 10 10 kids that they've got in in their home and this is not totally abnormal for home school some of these families are quite large then as we iterate through them every time an enrollment tries to pull its student attribute it's going to do a new database query this is called the n plus one query problem because you do one query to fetch the collection of things and then as you loop over them end times you do another query for each of those so the way to address that problem is to tell django hey while you're getting the enrollments go ahead and also do a join in sql to pull from the student table and pull the student information and django will do a join query to get both of those chunks of information and merge them together so that way when you do enrollment.student it doesn't have to go back to the database so it's a single query instead of multiple queries okay so we've got our enrollments and the next thing to do is to say um what do we want to say we want to say i said this was an object and it's going to be student of student and we're going to do for enrollment dot student in enrollments i really need to fix that highlight color i can't i cannot see the r on this with this line that i've added i switched to a dark theme not too long ago and there's still some adjustments that need to occur um wow that's just not even close matt it's enrollment so we need to loop we're doing a list comprehension and there we go and get my brain working sorry about that so now the test should pass assuming i did it all correctly okay so we've got our basic structure and in fact at this point we could flip over to the um the the template and render that uh the other aspect here is the the way this is structured is that it's going to show um whoever enrolled first so whoever has a lower primary key is going to be the the thing that the student that appears first on the list and that may not make sense um for the actual application but i think that's how i'm doing it elsewhere so we're going to stick with that and not worry about the ordering right now maybe there's some future where the ordering gets more specific to like alphabetized by name or something to that effect maybe there's a another layer of ordering here that the school year the the really the the right answer is probably the school year displays grade levels in a certain order so it probably makes sense to order by those grade levels but we're not gonna mess with that aspect right now i think it's going to naturally do the right thing for most circumstances okay so next bit i said that we needed in the structure is the the actual courses and the courses was going to be a another list so we'll take this and we'll extend it we'll add the courses and we'll just do an empty list at the moment to continue on the test driven development style and it should fail because there's no key there in the dictionary all right and and this is another challenge and i've actually experienced this at work recently is like dictionary like when you start to get to these complex data structures you start to see the limitations of pi test when it's ability to compare and like this is okay there's i guess this is still small enough that you can see everything that's in that dictionary and everything that's in this one but if the dictionaries get too big that starts to get annoying and hard to really observe the difference but i think it's still a reasonable size so the next thing we want to do is to take the simplest next step that we can and say courses is an empty list and that should get that test to pass all right so now we're on to the next piece of this now i need to go find the method that i expected i think it's on the student model so let's go actually the term that i'm looking for is i believe ordered courses oh it's uh not where i expected to be it's on the grade level well rats that makes sense because the it's not really the student isn't the owner of the course the student happens to be a participant in a course but really the student's participation in a course is via the grade level so we probably also need to come back let's close that to give ourselves more space we need to select the related grade level as well so we'll do that here we'll put in grade level into the select related so that's also going to pull grade level information out of this and when we get the grade level objects we will have that so the next thing that we want to do and and i'm starting to see like our our list comprehension i think is going to break it break down because we're getting a little complicated in the data that we're pulling out um so we're going to work from that in a moment but we'll continue to try and jam it into the list comprehension and just till it just doesn't fit anymore so we've got the we now have the gray level and we can get the ordered courses and that's going to go into this list but we want those to be a dictionary like i said so now it's time to get in this in to another step and in order to prove this out we're going to need a course on the grade level so let's create a course and we'll say course factory not the course resource factory just the course factory and we need to say because it's a many-to-many we need to say grade levels in equal enrollment grade level it gets a little weird with the many-to-many relationship on building the right data in the tests that's an unfortunate reality of the way all this is modeled but you do what you can do so we're going to create a new dictionary and the first thing that we want on there is not source it's we want course and we're going to pass in the course now we have the student and the course for the grade level that they are part of and if we run that test it should fail which which it did okay so we can now come down into this and we can say uh we know we have the grade level we know we have the enrollment so we're we're going to do this is where it's going to get ugly and this is where i'm not wild about this so we'll put it out as a list comprehension first and then i think i'll refactor it to be a for loop to kind of reflect that it's a more complex data structure that can't cleanly be put into a list comprehension so we will here say oops wrong side of that list looks like this so we're going to say course and course and then that'll be the close of that then we'll say for course in enrollment grade level get ordered courses now you can start to see like that just got ridiculous um although black black makes the formatting appealing to a degree but this is going to break down when we have to get the expected date i promise you so like even if you're like oh that doesn't look so bad and i think black was being really generous with how i formatted this um it's going to break down the next step is my my prediction oh and i got a failure so what's going on [Music] so we've got course we've got this is the data that i expected on this side and it maps to the course that we created the student is definitely enrolled in that grade level because we made an enrollment specifically wiring those together [Music] it's possible did i mess up i could have messed up how this factory is being built let's find some other examples of the course factory with grade levels because i know i've done that many times and it could just be that i'm messing it up yeah yeah yeah so look at the rest all the rest of these are saying it's actually not using the in operator i goofed on that so we just need to say grade levels all right and it was feeling a little wrong when i wrote it but i was willing to let it break and see what happens okay let's try the test again hopefully that's correct this time all right now that's more what i expected okay here's where we get back to the chunk of code i haven't even talked about the refactoring we did last week and the code bit that we added that will help complete this because there's a chunk of code that does all this calculation and we move that chunk of code to a file called the forecaster forecaster takes a student and takes a course and it figures it out figures out everything you need to do to get the end date and this was a complicated chunk of logic you can see it's not a super long file it's only the whole file is 71 lines but there's a lot of like weird little edge cases in here that have to factor in school breaks and all these other bits it's this kind of like what that's kind of annoying so um there's some stuff we need to consider here i'm already seeing performance problems uh sneaking in here that are gonna do a bunch more calculation than we want to do um that's a little disturbing so we'll see what we can do about that um the the biggest thing is like okay let's let's take let's think about this this forecaster has to do this computation and it passes in the student and the course but once you get into this method that's building um this list of stuff and figuring out the dates from the course we pulled the school year from the school we year we check on all of the school break models and those are just a couple of pieces and then we go and we pull from the course task table there's a ton of data that is coming into this and and that's not even counting the graded work like the query count for this could be ridiculous if we if we don't do the appropriate joins to get the right data so i think what i want to do is put in the dumb version of this and then we're going to put on the debug toolbar and we're going to see all of the queries that i anticipate are going to be in there and just completely bonkers broken um and that should give us something to work from as a target say like here's how bad this could get and here's what happens if you let your queries and methods be unplanned and get away from you but you know that's stuff that you don't notice unless you're like specifically paying attention to performance and performance matters because you know once you get a bunch of users on your site they're going to care and if you've got bad pages that are popular pages i don't know the popularity of this page but you still don't want pages performing really badly okay so the next thing we want we want the expected expected end date and i think we're going to have to do a few things to make sure that this is consistent we need to create a single single task uh so there there is an expected end date and well i guess that's also a missing test that we need is some of these courses uh that was going back to the forecaster the last forecast date that is the thing that we're going to be pulling from is an optional thing because there are a couple of circumstances where there's actually it's not actually possible to get the last date and so that's going to be a challenging aspect of of this is like well we have to factor in what do you do when the expected last date has no value so we'll have to consider that as well all right i think i'm just i'm trying to think through next steps i don't want to rush this in my head um about how to go about this okay next step i think is to change the test so we have the expectation so we want to set this thing up such that there is um a date that that's gonna appear and let's just do let's do the easiest case to pin this down is based on when a user actually completed a chunk of work so to do that we can create a coursework and coursework has a completed date associated with it and we can say that the we can say what we'll say the coursework factory and then we want to we're not going to directly create the course task but we will associate it with the course that we just created all right so i guess we'll also need to pull on the coursework factory which is from the students area so right here we'll say coursework factory whoops all right so now coming down to our expected data structure we'll say um uh what should we call this let's let's try and keep it simple last date is the name that we could put in the data structure um and i think that maps well with what i call the forecaster it says last forecast date yeah yeah yeah we'll we'll use that just to be super consistent so last forecast date and then we'll say the other side is going to be um the coursework completed completed date completion date i think it's completed date yeah it's completed you can see it right here so essentially we want to hit this this line right here is what what's ultimately what we were going to go for in this particular test and this is going this test should fail so i think the next thing that we can do this is where we have to are going to have to refactor the the actual view code because we need to bring in a forecaster now and the forecaster something that we'll have to create one of pass it the student in the course and say give me the date and we'll have to do that in a loop which we won't don't really have that opportunity with a um a list comprehension here so we'll start by um how should we start we'll stay we'll say students is a list and then we will say context students is equal to that variable that's the thing we're going to put data into and then we're essentially going to rewrite this chunk of code right here to give ourselves what we need so the outermost loop that we have to go over is the enrollment loop that was the first part of the comprehension so we're going to go through each of the enrollments and maybe this uh it probably would have been better to like refactor this to this first before modifying our test behavior but that's okay um so we're going to need to append to the students list uh i'll call it a student info because that's kind of what it is it's a collection of it's really not it's a primitive it's a dictionary that has student and so that that's going to go in there so we need the student info which as i said just a second ago is a minimally it has the student wow i can't even read that because it's just red on red that's brutal okay which i spelled wrong so i wouldn't have not seen that so that's going to be the enrollment student so even getting this far we're kind of getting back to the data structure that we already built out it doesn't have the courses anymore so that whole key is going to be missing we're missing the inner comprehension and that's the piece that we need to build in a second level of the for loop so we've we're essentially unrolling the list comprehension into something that we can use so now we want to say for course in enrollment grade level get ordered courses and then do do what we want to take the student info and set the um well nope not quite so we we need to put courses in here and say that's going to be a list and so we want to append to this course info or this courses key we want to append a new dictionary and that's going to be a course info again i'm calling it info i don't know where i devised this particular pattern for our name it's not not fantastic but um but it works that to kind of indicate that it's not a course object it's not a course model object it's something else it's a it's a course model object plus some other stuff okay so we need to say we actually need a course info then which is again going to be another dictionary and now we're getting back close to where we were to get parity okay so this test is still going to pat is sorry it's still going to fail apparently i ran the whole file that time and it's going to fail because we're looking for the last forecast state it's not in there the rest of it should be fine though but here's where we can bring in the forecaster and actually do the rest of what we expect in fact at this point i think we've done everything that was in the old comprehension so i'm going to delete that code and this code is starting to look pretty darn dense so i'm going to give it some breathing room just for my own sanity um yeah that's probably okay just kind of give it some space not strictly necessary but you know just like white space is good in a website design there's there's is some benefit to having vertical white space between and having blank lines between stuff to kind of visually group things together some people prefer like ultra dense code but i think that's i don't think that's the best so we have the course and in the course info and we're going to have the um the other piece here is the last forecast um yeah forecast date is what we called it that's the key and that's where we called the forecaster that doesn't exist yet so i'll just believe that there is a red thing and we'll say forecaster is equal to a forecaster that takes the student and it takes the course but that's going to break because there is no student directly you got to get through the enrollment and we need to get the forecaster from the the other import that we haven't done yet so the forecaster is from dot forecaster import forecaster all right still haven't used it though so it's still complaining it's also telling me oh oh oh i guess maybe we didn't give that as data all right let's bring up the forecaster clearly we didn't do an init method all right so we just create it and then use it so let's delete all the inter inside where did i go there let's delete all the inside of that stuff and then we're going to say forecaster get last forecast date and pass the enrollment student and the course now if we run all the tests for this file boom we actually get what we expect so the last forecast date corresponded to the last to the coursework so this is actually really good this is good we're ready to move on i think to the template and um yeah check it out see how it goes and then start to see how bad the performance is because i'm predicting it's going to be pretty bad so let's return to that forecast html file or cast html so step one here trying to think if i want to bring in another table first and then modify that to use the data or if i just want to see the data yeah let's just see the data for now and see how bad the forecasting is and then then kind of clean up the design so we've got a loop and the outermost loop is for student in or actually i guess it's the student info in the students that's the first thing we've got and we can print out uh we'll just put a bunch of paragraph tags for all this stuff we'll say student info dot student that should do get the string representation for the student which is their their name and the server's been running this whole time so we should be able to refresh this and yep we got my kids names which is the test data that i have for this account all right so that's layer one and then we go to the next layer and we say for course info in student info dot courses and we will print the course course info course and let's also go ahead and put the forecast date why not so the course info last forecast date and i mean as like we just did all that work basically for two for loops in a template um that's how it goes all right so wall of data but this is exactly what she's looking for i mean there's some cleanup to do like it's not possible to have a date on some of these things but this is it all right now let's go see how brutal the actual performance is it may not be the end of the world it may not even be worth optimizing that much it depends but it could be pretty eye opening so we got the debug toolbar i was waiting for that to load yeah 197 queries that is terrible i would expect this to be well a number far less than that i don't i don't know and and since we're working locally here this is you wouldn't you don't even like really notice um but when you have an actual website and you have like a family there's going to be that family that if if the site exit is successful and you're around long enough there's all you're going to find these edge cases these outlier groups and there's going to be a family with 10 students and each student is doing 15 subjects and each you know like each one of them has hundreds of tasks and like all of these things add up to be this just explosive number of queries that that has to happen and we're looking at 197 queries for two students which is just crazy crazy crazy so what's happening is what i feared is that the the forecaster um is not very efficient for this page now here's the weird part though if we go look at the students or if we go to the the week view and um so that's going to bring up all of the student information and then we can go from that page and look at a specific course um and then see how many queries are happening on that one so let's pick for example uh math2 here all right and let's look at the debug toolbar 10 queries now why is that so the way this is so you see all these dates here this comes from the forecaster the refactored code and the the thing that we're looking at on the forecast page is this very last date so if we go over to that other view and check out math2 it should say uh july 16th in fact let's do that just go see if we're just to kind of prove to ourselves that it's right so it's running all these queries right now it's a lot it doesn't understand why you're doing this to it there we go july 16th okay so what happened we added some select relateds back on the view but it wasn't good enough um we got the select related as here in fact we can make this even worse let's let's make it worse just to see how bad it can get so we'll take that out we'll do that and so we've got 197 queries now well it's only going to probably increase by maybe four or something like that we'll see still loading yeah it increased by four as a one i was able to do that math because there's two students and i had to do a query for each student as well as each grade level for the enrollment so there's four additional queries so that part's not so bad so it seemed like the optimization that we did of doing a select related here just didn't cut it and it's it's true it didn't it's because this get ordered list of courses does not that gets passed in here the there's so much data that is pulled out of this course the course is used as this anchor point that pulls out all this other stuff that we saw and what we can return to the forecaster to to prove that and as i said you know it had to do it goes does a lot most of its work the actual forecast date method that we're using goes through all that task processing and does the full list of tasks that you saw on that other page and then just checks the last one so it does the kind of the stupidest approach possible maybe there's a smarter way to do this that bypasses all that computation i'm not really sure but really what we want to get into is what can we do to optimize and perhaps prefetch the extra data that we're going to need for this stuff so one area that this really breaks down in is this this access right here on line 32. so every time we're doing a course it's accessing the school year but nothing about this get ordered courses said you need to go ahead and have the school year pre-fetch the school year so there's there's one and then once you access the school year and give it a date this is break thing has to do another set of queries and we can probably see some of the evidence of this by looking at the debug toolbar so it gets the courses gets the gray levels and where does it start doing other stuff so it's got to do this school break that's from the school year that is break method that we saw let's see if it shows up um some of the challenge here is you got to filter out which stuff belongs to g unicorn and which stuff is actually from your code yeah here we go so this boolean is break method actually has a lot behind it it needs to check the breaks and before the breaks there's a cache that gets created for every school year instance so the first time you ask if you're asking about on that request uh what what breaks are available for the school year and and breaks for the context is like vacation days when are you going on holiday uh for your your family um and it needs to be able to answer that so rather than answer it one at a time i took the approach of just say give me all the breaks and let's shove them all in a dictionary and cash it for that school year instance but the problem is when course that school year happens django is doing a fetch for the school year each time and it's not a shared cash so even though the school year for the all of those courses is exactly the same each one of them has their own cache of breaks so all of the breaks for the school year have to be fetched over and over and over again so really we want to go in there and say hey look here is here's an optimization just do one school year so i don't remember i think here there's a problem we're going to encounter i believe so let's go to the grade level and look at that problem if we go to the uh where is this model i think it's in the schools school models so one of these i think is grade level all right so here's the problem this is not a query set it is a list which means that the database access has already happened by the time that this method runs it's really unfortunate so i'm trying to think of like what we can do to optimize this path one thing one thing we could do is replace the school year attribute ourself i don't know that seems like a crazy idea but it may be worth it and we might want to do it this might just be an optimization that we want to put in in this method in the get ordered courses because knowing that i don't have the access to the school year here it might make sense to go ahead and get the school year for one of them and assign it like after after the list so what would that look like um this and this would reduce a huge number of queries i think so let's let's try this i don't know that this will be successful but we'll try so we'll change this to courses as a list and we're going to um we're going to return courses after we're done and so essentially we're like eager loading a chunk of code we're saying like this is so often going to be used the school year attribute that we'll prefetch it and stick it on all of the courses ourselves so the grade level we're already in the grade level model so we'll do an access to get the school year or really what we can do is we can say we can't do this on a comprehension so we'll say four course in courses and then we'll say course school year is equal to the self school year so i'm hoping what this does is it like will replace django's attribute access so that when django tries to fetch the school year for the first time it'll say like alright i already have an instance here i don't need to go do an additional query because we have taken that and deliberately attached that information here and i'm trying to remember how how how the school year is able even able to be attached to the course is that a property of the course model because i don't that's not something that's actually part of the model so it's probably just going yeah look at all the stuff that it's doing wow wow wow good gracious yeah this is how ridiculous it gets so we already have the grade level fetched and getting the courses and we're going to the school year property the school year property is going back and it's going to the many to many relationships so it's actually going back up to the grade levels so these this sql number of sql queries is starting to make sense so before here right here in this chunk of code i think the when when the course is trying to get access to its own school year it is going back through its set of grade levels and looking up at the school year so it's like doing this whole weird dance where in this ordered model method we already have the school year or we already have close access to it we know it's coming from a grade level we know it's a real school year so it makes zero sense to go through this cached property all right so i'm afraid though that because we're going through the cash property that django is not going to be happy about that it's going to probably say like you can't do it's a it's a property and not a setter so it's probably going to say you can't do this so let's try it out see what happens no i was wrong so we dramatically reduce the number of queries it's still a lot still 125. that's a ton but it's not 197. so we knocked out one category of errors not they're not errors they're just performance problems by taking this eager loading step of putting the school year directly on the um the course and well i don't know this part probably doesn't matter but let's do one minor thing here this is probably a micro optimization that really isn't going to be measurably different we can so every even in even attribute access requires some extra processing uh it requires essentially a dictionary lookup so if we assign this to a local variable first and then do this in a loop then that attribute axis is not going to be there that probably is almost negligible impact but we can it's just while we're here we can make that change all right so 125 queries where do we go next is there a way that we can continue to optimize this how much do we need to um let's let's look at that next because i would love to get this down to a reasonable number where these n plus one query bugs which is you can see happening here whenever it says 38 so 38 must be the number of courses that we're looping over and so it's doing this 38 this it's django debug toolbar is detecting as a pattern that we're doing this 38 times and that is what we what i'd like to try and eliminate i don't know that we can but because you know it probably can't for some of these things because they're we have to ask the question 38 times like what is the end date for each of those courses there are 38 of them it's unavoidable so we need to at least go through it once but maybe the thing that we can optimize is if there is a set pattern of these are the queries that we run for each of the 38 each of the 38 can we reduce one of those can we find something that we could optimize in there and the answer might be no not easily in which case we'll just say okay this is going to be how it is and it won't be the slowest or excuse me it won't be the fastest page on the site but it's a page with a lot of information and calculations so maybe that's okay and the performance might be reasonable enough for most cases okay so let's let's see if we can see where the pattern is so we probably want to focus on where it starts with this number of 38 similar queries and see if we can find the there's got to be like a loop of these so i see students coursework students enrollment course course task student coursework student enrollment course course task all right so that's the loop um it's got to do that for each of them we can maybe dig into which one's doing which we're pulling from the enrollment table again well let's see let's see what this can tell us maybe there's more information about the code where it can find it can highlight this pattern um so let's start the coursework what is actually calling this query and what is it trying to do it's so the first thing it's trying to do is get the coursework actually just go down the code see if we can find it yeah okay so you have to get all of the coursework for the course that's not avoidable that has to be done because it is um it goes in into the computation because if a student has done some of the coursework it affects like when dates complete and so on so that's that's there and then it actually happens at this line in the get course work by task but maybe maybe our the method in this forecaster is not optimized on its own so let's look at that for a minute i've got this big loop that's doing all this stuff and here's where it's doing that first query because it has to populate this courseworks by task dictionary so that it can do lookups to say is this done or not okay i think that's a necessary piece and it's a piece that we have to ask for each course so hey tom tries welcomes um we're doing some optimization work at this point um got any questions feel free to ask so the only thing i can think of is perhaps we could when fetching the courses also fetch all of the tasks is that even reasonable no not really we'd have to like fetch it'd be a pretty monstrous join so i don't even know if it's going to benefit that beneficial so you can see here we're doing the coursework and getting the task at the same time so we're at least doing that optimization in one step we know we need to get all the coursework to do some of this computation we know we need to loop over all the tasks so that's done in one step that's good or it appears to be maybe it's not actually in which case we need to rethink something so that's that first query and that's why you can see there's a join that's in there's an inner join here and it's getting the students coursework and the courses course task all together so that's what that first query does all right next what is this guy so interestingly it's doing another query on the enrollment and grade level that seems wrong because we've already done at least from the point of view of this forecast view we've already done an enrollment lookup so what is this doing how is this getting in here okay so we have get tasks for course and selects graded work that's interesting let's look at that for a moment i think this might be another one where it's like an annoying thing we've got to do here is this enrollments by course cash hmm so i think this cache is not being populated yeah look at that so we actually so there's a get the active courses for the school year is using this cache get tasks for course is trying to pull from the cash but it's only the cash is only being populated in one spot so what's happening okay so this might be a spot we can optimize maybe because the in in the view that we're working with we have the enrollment so in theory oh but we don't have the courses yet sorry y'all i'm just trying to think through some of this stuff and what it's doing this is the part right here that makes the um that makes this hard is that the students the student grade level the tasks that they can be in can come from either tasks that are just about the grade level specifically or i'm not i'm not saying that very well a a course can have tasks and those tasks can be for a specific grade level so like getting back to this example where my kids had a shared art course well maybe my son who's a little bit older has to do some extra studying on some certain aspect of the art it would be possible to add a task that was specific to his grade level that was higher that my daughter doesn't have to do and that's why this logic is in here and so it gets a little complicated and that's why this method just has to be here just like this is how you have to get the tasks because it has to factor in um that scenario you can't just query the task table directly so the challenge is um because i'm not like linking together a bunch of query sets there's some extra querying here which is why i have these caches on here but this cache is empty for the students and it's only filled when asking for the active courses so that's one layer of queries somewhere through here that's this enrollment and grade level set right here so 38 of these queries these join queries are coming from doing this check on the enrollment table over and over and over again when it doesn't need to all right so let me think if there's a way we can get into this so it's telling us line 287 get tasks 4 287 yeah it's coming in here in and it's a cast a cache miss essentially so what can we do if anything to make that better is there can we populate the cash i'm kind of getting myself convinced that this may not be worth the effort right now like there there could be like if the app was ever wildly successful and this page was showing up on a performance monitoring tool that like this is the worst page on the site well then sure but uh but i i think we're just kind of i'm working myself into not a corner but just saying like this is for the level of effort that we're getting out of this it's it's not very evident that it's going to have a huge payoff so let's say okay that is a cold cash that we could fix but maybe it's not worth it so let's leave that one alone all right so why is why are we coming back for this course task and graded work i thought we already did that oh okay we did do the we did build the query set that that is happening it's down here um but it is not being processed in the loop until after this line so this is this call is calling this private method that gets the query set and um and then it does that in fact maybe i'll just reorder this because like well i guess it doesn't truly matter or actually wait a minute wait wait wait they're two different kinds of relationships i'm sorry they both are necessary they're both are different so one of them is a map of here's all the coursework you've already done the coursework is like the actual work that the student has completed and so it has a set of tasks to to kind of overlay onto here's all the tasks total here's what the student has done and there's a union that happens there and of course if if the student has done everything in the course the union is full like it's a it becomes a full merged set together but we don't know that until at this point when this calculation is happening so both queries are required and just because they do different purposes all right so i think i'm kind of explaining to myself that that's necessary the the one optimization that i see is that we could further take on if we wanted to continue to refine this is to warm up that cash for the student for enrollment i just don't know that it's going to be a big problem and again we're i'm not doing i'm not really doing a lot of performance monitoring on this and this is a page that will not be the super most common page to use on the site so i'm not really worried that it's going to tank the performance of the whole site so i think getting it down from 197 to 125 with those couple lines change that we did on that ordered course thing that felt like the right thing to do and you know where was that like this this change right here let's actually let's go back and comment that on the school's model and that that should be sufficient for now so we'll say eager load the school year and to avoid performance hit of hitting um of hitting the cached property on course in a loop okay i feel good about that i think it was the right change okay i think that sort of brings us to nearly to the end of what we need to do now the rest of this actually let's do let's do one thing let's kill the server for a minute let's let that run while we chat so it's going to check that all the tests are right i think the last thing you're really willing to do for the rest of the stream is go in and take that couple of for loops that we have and make a table out of it and i'm probably will borrow from a pre-existing table um that in fact that that one we were looking at that had the plan dates might be the closest thing we care about because it already had a format it was pretty close anyway it had a format that showed the student you could use that as the header or a you know h2 tag and it had the table it had a table of tasks but we could repurpose that to be a table of courses and list out the dates so let's see that all of our tests are still passing good they all are we've got 100 coverage i think we're ready to to move on well it's curious i don't know if this stream or what that's that was way slower than the past not sure what's happening there something to debug later okay so let's return to this forecast view and the other one that we want to bring up is the student student course view well is that really the we might need to borrow pieces from other spots like we need to we need a header common header that we use and the best one that i can think of is from the daily page to actually get the student uh where are you [Music] yeah there's an h2 let's use the student first name that feels right i don't think that we need to print this one so i'm not too worried about that part but let's take this piece and let's put it in our template let's replace this paragraph tag and fill in change schedule here to student info let's take a look so i think that's all we need the daily template for restart and um see how it goes so now we've got mark and faye we don't need their last names in here and then we just need to fill in this table with something more reasonable so the next thing i want to pull from is this task items table right here that is not going to work initially it's got all sorts of information that has nothing to do with what we need the challenge with some of these is like they're so customized that i don't necessarily like i want the styling but all of the html baggage that you don't realize is necessarily under the covers is there too so let's kind of poke around and see if there's another place that is more appropriate we've got this kind of simple progress report page that's pretty good maybe we just need this because that is a very reasonable format this is just all the grades and stuff that they'll use all right i'm liking this one it's a lot less there's a lot less stuff there the the other one that we were looking at that i was thinking of using was that one on the weak view but what what's here that you don't necessarily realize is that all of these are clickable all those links over here so there's extra markup here that we don't necessarily need there's a bunch of hidden markup to jump back to stuff appropriately and i think that's just going to be overkill i don't think we need to well at some point those those courses will probably be clickable i'm guessing that'll be a pretty quick feature request after that but um we'll start with the simple and get the data out there and then she can my spouse can tell me here's what i'd like to see so this is the report progress report report um progress import page and we've got the table this is the table that we want right here and you can see this is much much much simpler it's like 10 lines of table code i mean it's really not much at all just plop it there and we're going to take this name of task we'll change it to course and we'll change score to uh i'll call it forecast date for now or maybe last forecast date like like the variables called and if she doesn't like that name she can tell me and then we've got the we actually need to populate it now so what goes in here well that's the overarching table like i guess we need so we don't even need to like loop over this the way it's being looped over it's actually unnecessary so we need to drop this for loop into this level here and get rid of the grades thing get rid of the end for here and we'll get rid of that paragraph tag in a minute so i moved the tags a little bit that they'll line up and then we can go to grade here and we'll change around the that stuff and say um course info course that's the first bit and the second bit is the course info last forecast date we need another template tag there we'll get to in a second because we have none uh for sometimes so that needs to display something appropriate okay well still stuff to clean up here let's resize the flex box to give it some better a more appropriate sizing um we'll switch it from five sticks so these courses they don't have near the same space requirements so the the other table was like tasks and stuff and the task names can be quite a bit longer than a course name typically so i think we can change a few aspects of this we can bring in the dimensions quite a lot we can say instead of max 2xl we'll say that the whole thing is a max xl that brings it in maybe even that's well let's get the the spacing of the columns right first um i'll keep it on sixth just in case i ever want to change it quickly because switching into thirds is then you can only work in thirds but for the moment i'm going to do 4 6 and 2 6 and that should switch it to thirds but with a more granular control so boom that was what i was hoping to get out of that the last forecast date becomes a one line thing and you can see we've got none and that's the thing that we need to change um so we want ooh there's some funky stuff in here well i'll get some my problem is my test account has like a bunch of stuff that i'm unsure about now so i don't know if this double science entry is because i put two different science classes in there with the same name or if it's something else but what i want to do is say instead of none here say like unable to forecast and so that attribute right now is literally a none so we want to go to django's built-in template tags something i search for all the time so it's nice and conveniently in my search history and we want to go and use a tag called i think it's this one default if none so this is a filter that we put on something and we can say get rid of the progress report we're done with that and now we can go to the forecast date and we know that this could be none so we put a filter it says default if none and for now i'm just going to say unable to forecast yeah let's leave it at that see how it fits don't need that don't need that unable to forecast okay great this is the default format for the date which i don't think is consistent with other places that i've used in fact i just closed one of the history yeah so it's really in this year month date format so let's be using that so that was on the school course page i know the student course page and so there's probably a date yeah ymd so we're going to go to oh junk all right well so because of that we can't really do the date filter i think because we've already got a filter on there the default this default if none can you chain those together uh god that's a good question um django default if none date filter i don't know maybe you can change the order of filters okay you can put the date first we'll find out i should have just tried it so we're going to put in front of the default if none we'll say date is equal to y m d or default if none of unable to forecast whoops not what i wanted okay rats [Music] hmm that stack overflow had default so i wonder if that will change it oh that doesn't make me feel good why does that work django use default aho that's tricky tricky look at this little hint here so date the date filter will return an empty string if it's not a date time or a date object so it's taking advantage of that to chain it together it's return so really what it is doing is it's returning something and passing the return to default and by the reason it wasn't showing anything with default if none is because it wasn't a none anymore at that point was an empty string [Music] all right i'll i'll let that slide right now i'm not i don't not really wild about that part but okay now let's try putting a text center on here to see if it looks any better than it does right now because that label is a little long no that's just weird because this is so much longer yeah it's undo that part so we'll live with a little bit of left side alignment there that even though it doesn't quite match up super well it's okay and maybe we should just do 50 50. let's try that so let's replace all of these okay i think that reads better um and instead of extra large what if we change it to a large keep just bringing it down because i i don't think there's any real reason to make the user visually kind of like of course date over there that's really spread out that doesn't make sense to me and this is a really this is actually quite a long course name for from what i've seen especially compared to what you can see here but i just mean the general appearance on the school year so i anticipate that there will not be many courses that have a name that's are much longer than that next into the second line i'm okay if that happens [Music] yeah all right so here we go we got it this is the projected dates i think it it answers what is needed uh for this particular feature and it can be extended but we don't have a need to do that yet there's probably some future here that i need to explain what unable forecast means uh but uh right now i think that's it's okay to leave it as as is a kind of a minimal and i think this will probably make her pretty happy so let's review what we've got here we've got we made this change to get ordered courses to eagerly load in the school year to try and make this performance optimization we wrote our test that has the context and and ultimately we're not really testing a lot um here like the the actual test to do everything we need to do it's only a handful of lines of additional tests to an existing test to assert on the context so that's kind of nice there's probably some stuff that we could test better or more by putting more in there but i don't really feel like it right now this is sufficient getting down to the actual view code we did have to make a bunch of stuff here we had to change that comprehension into a more complex loop um that is building all of this data together there is an argument to extract this into the forecaster i don't know i'm not really sure that that's necessary if if you joined in last time the way that i did this is i took logic that was in a view and put it in the forecaster and that forecaster then we could use that in this view and now we're using the same forecaster again so there's there is an argument to be said of well what if we flip this on its head and say hey forecaster give me all of the last forecasted dates for all the students courses and here is the student and the school year that you want to check out in that case maybe there is some optimization that could be done or just regrouping i don't really know but i'm i'm not going to try and over optimize the problem of of make this perfect and then we've got this little chunk of template code and that was really all that we had to do to make a kind of simple display that shows exactly the information that my user is looking for and i think it it we've we tested it out it seems to do what it needs to do um so that's i'm gonna call that the first version of this feature that i'm okay shipping and um turn that on for for her and i think that's probably going to bring us to the end of the stream i've been going for nearly well coming up on an hour and 45 minutes and we've been working on a single page hopefully um this was instructive to you there's there's a lot that goes into making that work i will take this content and turn around and put it up on youtube on this on the channel you can go find me as matt layman on youtube and i've got videos from all of my streams up there um even the really really rough ones in the first one so uh if you go all the way back to the beginning it's come a long way i'll say i'll say that much so this content will go out and probably go out by tomorrow hope you found this useful if you did please give me a follow or like or subscribe or whatever platform you're viewing this on that would be really helpful and i hope you have a great night take care you

Channel: Matt Layman

Views: 282

Rating: 5 out of 5

Keywords: games, Python, Django, SaaS, TDD

Id: Phk-yV-yjjA

Length: 96min 54sec (5814 seconds)

Published: Thu Mar 18 2021