Django Database Transactions / atomic() function

in this video we're going to look at database transactions in D Jango these are a key Concept in relational databases and it's vital for backend developers to understand transactions what we're going to do in this video is cover what a transaction is and we're going to look briefly at the concept of atomicity we're then going to quickly look at the default jangle Behavior with transactions which is to autoc commit and then we're going to set up a simple problem that uses a product and an order model where a product's number in stock should be updated whenever an order is placed we're going to see how we can use transactions to enforce that consistency between the two models and we're also going to see some problems that can occur when we don't use transactions and our database State can be inconsistent and once we've worked through that problem we're also going to see how to perform certain actions when a transaction successfully commits to the database for example you might want to email a user when an order has been successfully placed but not before that transaction has committed so there's a lot to cover here let's let's get started now I have the Jango documentation on database transactions open here we will refer to this documentation throughout the video but firstly let's define exactly what a transaction is and I'm going to go to the postgres documentation on transactions here I leave a link to both of those pages below the video and as it says here transactions are a fundamental concept of all database systems and the essential point of a transaction is that it bundles multiple steps into a single All or Nothing operation and the intermediate states of a transaction between the steps are not visible to other concurrent transactions and that means if you have an operation that updates multiple values and you have a concurrent transaction they are not going to be able to read the temporary values before they are committed to the database so the transaction has to complete successfully in order to write the changes to the database and if any single step of a transaction fails there will be what's called a roll back and the database will be rolled back to its previous state and no concurrent transactions can then access any of that data that was changed before the roll back so transactions are an All or Nothing operation we're going to see more of that throughout the video now there's an important Concept in transactions called atomicity and I'm going to go to a very helpful blog post on this topic acid is an acronym for Atomic consistent isolated and durable and these are all pertinent to transactions but the definition of atomic is basically what we just saw in the postgresql doc dentation atomicity in the context of databases means that you either commit the entirety of the transaction that's occurring or you have no transaction at all and your changes are rolled back and I'll leave a link to this page in the description you can read about consistency isolation and durability as well if you want to know more about how relational databases work I'm going to go back to the jangle documentation and we're going to see how D Jango manages transactions by default here the default behavior is to run in what's called autocommit mode now in autoc commit mode each query is immediately committed to the database so when autocommit mode is on each query is wrapped in its own transaction that will automatically commit the changes to the database at the end of that query if all goes well now we're going to write some code in a second but we're going to go back to the post SQL documentation here and if we scroll down you can see some of the keywords that are used in SQL the commit statement in the context of this query here we begin a transaction with the begin statement and then we perform an update to a table called accounts where we set the balance equal to whatever it was before minus 100 and we do that where the name of the user in that accounts table is Alice and then after performing that operation we commit the change to the database and that is what actually tells the database this is the final change we want to actually make this change in the underlying table and we want to make it persist to the database now if partly through the transaction we decide we do not want to commit for example Alice's balance might be negative after subtracting 100 here then you can issue a roll back command instead of commit and if you issue the roll back here instead of the commit this statement here will be rolled back and the balance of alyses account will be the same as it was before and in autoc commit mode jangle will automatically commit every query that we run to the database and that's the default behavior in jangle so with that said that's enough talking from me let's actually write some code now and go to vs code here I have the orm series project open and what we're actually going to do here is go to the models.py file and we're going to Define two custom models that are going to be used in this video to demonstrate the concept of transactions so I'm going to go below the sale model here and actually these models are kind of nothing to do with the other models that we have in this project this is purely for the purposes of this video although we will use these models again in later videos so I'm going to define a pro product model and I'm also going to Define another model called order so let's start with the product model here I'm going to inherit from jango's model class and then we can Define some fields on that model now I'm going to paste these in just so we're not taking up too much time here we have the name of the product that's a car field and we also have the number in stock which we're setting to a jangle positive small integer field and we have a dund string method returning the product's name underneath that I'm going to define a second model called order and again that's going to inherit from the model class and we're going to Define some fields on this order model now this is going to be a very simplified version of an ordering system here we're going to have a foreign key from the product or rather from the order to the product here and when we delete a product let's set the on delete parameter to models. Cascade so we have a product linked to an order so an order can only be for a single product which isn't ideal for an e-commerce system but this is a simplified version of that we also have a field here called number of items and we're going to set that again to a positive small integer field so the idea behind this primitive ordering system is that we can have an order placed which is linked to a single product and that can have a certain quantity or number of items so let's say we ordered a book we may order that single book and that would be attached as the foreign key and we might order five copies of that book which would set the number of items to five now this number here is the number in stock and we want to update that whenever an order is placed but this is a contingent operation the order has to be successfully placed and once it's placed we then want to update the number in stock we cannot create the order and then the system will crash without updating the number in stock because then the database is going to be inconsistent so we're going to use transactions in order to tie this together and demonstrate how to do that with Jango now a couple of last things to add here in the order model of course in the real system we would also have a user model the order is presumably placed by a user so we need to track those details for the user and of course you might have another model called order item or something similar and that would allow a single order to have multiple items in that particular order we're not going to do that here but just to note that the final thing we're going to do is create a Dunder string method for the order model so let's paste that Thunder string method in here and the Thunder string method refers to the number of items that been ordered for the particular product's name so now that we've done that in the terminal I'm going to run a couple of Django commands python manage.py make migrations is going to create the migration file for these two new models and then once we've done that we can run migrate that's going to change and add those tables to the database now throughout this series we've been working in the orm script.py file but we're actually going to create a front end in this video that's going to allow a user to place an order via a jangle form so let's make the terminal smaller here and I'm going to go to the apps forms.py file we defined this form earlier in the series I'm going to remove that and we're going to define a new form now from the models module we're going to import the order model that we just defined and we're going to define a form in that called Product order form that's going to inherit from jango's forms. model form class and then within the model form we can Define The Meta class and Define the model that this form is linked to which is going to be the order model and as well as the model we're going to define the fields that we want to show in the form using the fields property of the meta class we want to show the product forign key and we also want to show the other field on that model which is number of items now we need a view that's going to serve this form so I'm going to copy the name of the form and we're going to import that into the views.py file so let's at the top here from the cord. forms file we're going to import that model or rather that form class and then I'm going to define a new view just down below here called order product and that's going to take as an argument the jangle request object and what we're going to do is create a form instance here by instantiating that form and then we'll add it to the context with a key of form so let's do that and then what we're going to do is copy the render method here from the above view so we're returning the render method we're passing the request in here I'm going to actually Define a new HTML template for the purposes of this video it's going to be order. HTML and then we also need to pass that context in to give us access to that form in this template that we're about to create so let's now go to the templates directory and I'm going to create a new file called order. HTML within this directory and this template is going to extend the base template that we have in this project so it's going to extend base. HTML and if we look at base. HTML that defines a block called content so let's copy that and in fact we'll copy the end block statement as well and go back to order. HTML and we're going to paste that in here and within this block we're going to write some HTML code that's going to display the form on this page so let's use a form element and that's going to be submitted with a method of post so it's going to send a post request to the server when it's submitted and then of course since a post request is an unsafe method it can change data on the server we're going to also attach the csrf token for security and then we can use the form that we added to the context and we're going to use the ASP property to render each field in the form in a paragraph tag so it's not going to look particularly good on the front end but we're not doing any styling in this video so the final thing we need to do after we've defined that form is add a submit button so I'm going to paste in a submit button here or an input with type submit that's going to allow us to submit the form and send the post request with that data to the server now the last thing we need to do before we can go to the page and look at this form is go to jango's urls.py file or the one that we've defined within the core application and I'm going to add a new URL path to the URL patterns here and that's going to be to a URL of order and it's going to call that order product view that we just defined once we've done all that we can run the Django development server at the bottom and I'm getting an error here that's because I'm trying to import the form that we removed earlier on so from forms import restaurant form we can remove that we no longer need that form so we can run the server after doing that and we can then go to the browser and we have here the form on the page it contains two Fields one for the product and we currently don't have any products in the database so there's nothing in this drop down here and we also have a field to select the number of items for that product that we want to order and we're not seeing the submit button if I go back to VSS code that's because I've not saved this file so let's save the file and refresh the page and now we can see the submit button now in order to add some products to the database what I'm going to do is go to Jango admin. Pi file and I'm going to import the new product model that we defined and I'm going to also import the order model and once we have them imported at the bottom here I'm going to add two more calls to admin. site. register and I'm going to paste the product in for the first one and the order for the second one so those two models should now be available in the jangle admin UI so let's go back to the browser and I'm going to go to the admin UI now I have a user called admin who I'm going to log in as and then we should see the admin UI and we have these models at the left hand side what I'm going to do is add three products to the database so we can do that by clicking on this form and we get the name of the product and we can get the number of that particular product in stock in this imaginary e-commerce system so let's add a product called book and we're going to set the number in stock to four and we can save that and I'm now going to add a couple of extra products once we've done that we have four products we have a camera we have shoes we have a video game and we have have a book pretty strange store but let's go with this for now so now that we have these products in the database I'm going to go back to the order URL that we had here and we should see these products appearing in this select box so what I could do is select the book and say that I want two of that particular item and then submit this to the server but the problem is the server is not going to handle the post request at the moment so let's go back to the views.py file and we're going to handle the post request to this order product URL and don't worry we'll be getting into the database transaction stuff very soon but if we handle the post request by checking if the request. method is equal to a post request what we can then do is instantiate the form by copying this statement here but this time we're going to pass the request. poost data into the form and that's going to instantiate the form with whatever the user has submitted from the front end we can then check if that data is valid by calling the forms as valid function and if the form is valid we you can create an order by calling the forms save function and the form has a save function because it's a jangle model form and it's tied to a particular jangle model the order model by these two Fields the product field and the number of items so when we submit that data the form will check that it's valid and if it is valid we can then call the save method to save that order to the database now we don't just want to save the order and that's because when we create a particular order if we go back to models Pi that order is for a particular product and a particular number of items so what we need to do is also go to that product and update the number that's in stock for that product so we need to do that let's go back to views.py and we're going to write a little bit of code here that's going to handle that so once we've saved the order to the database we can access the product that was selected by the user by following that foreign key to the product and that has a field of number in stock which we can then subtract subract the number of items that were submitted in that form so we get that by looking at the order. number of items field once we've updated that we can call the product models save method by following that foreign key and calling save so we take the order that was submitted from the front end using this form here which was for let's say a book and let's say we wanted to select two of these what that's going to do on the server is it's going to save that order to the database and then after it saves that is going to update the number in stock for the associated product which is the book by subtracting the number of items that were ordered in that particular order from the number in stock that was in the database before and these statements should be done in a transaction we're going to see that soon but if we go back to models.py the key point to take away here is that this number of items should update the number in stock so the number in stock will be updated when an order is placed so we need a transaction to make sure that when an order is placed the number in stock will update accordingly we'll dive into exactly what that entails very soon let's close models.py and what we want to do when the form is valid and we've saved these objects to the database is we want to redirect the user back to the order page so I'm going to go to the top and from jangle do shortcuts let's import the redirect function and I'm going to copy that and go back to this F statement once we have saved those items to the database we can return a redirect and the name of the URL was or product that matches what's in the urls.py file for the named URL now the final thing we need to do in the view is we need to handle the instance where the form is not valid and we're going to add some Logic for validation very soon to this code let's write that else block just now where we set up a context here and add the form to that context and remember in the else block the form is not valid we've instantiated it with the data but the form will contain errors at that point so we've set up the context we also want to render the order form so we need to copy this render statement at the bottom and paste that into the else block so that is the code that handles a post request here if it's valid it's going to save the data to the database and redirect back to the order product page which will allow the user to then fill out a new order if they want to but in the case where the form is not valid we go to the else block and we're adding the invalid form to the context and we are rendering that ordered HTML template which is this page here and that's going to contain the data that was previously submitted but is invalid so let's go back to vs code and we're going to save all of this and we're going to test this out on the page if we refresh this page and we select the book and let's say we want to order a single book from this order if we submit that to the database we can go back to the jangle admin now and we can go to the orders in the database here and we have this one order that we've just submitted so the post request is now working it's saving the data for that order to the database but we have a problem if we go back to the product model here and we look at the book product where we just submitted that order we now have zero items in stock so if we go back to the page here and we select book again and let's say we wanted to order another book and submit that to the database you can see we're getting an Integrity error and the Integrity error is telling us that a check constraint has failed for the number in stock column now let's get back to models.py in the jangle application if we look at the product model the number in stock is a positive small integer field so this cannot be negative and the database adds that check constraint to make sure it's not negative the problem is when we submit the order here we are updating the number in stock by subtracting the number of items in that order and in that case we submitted an extra book there so it's going to try and subtract one from zero that would have resulted in netive - 1 and that is why we see the check constraint error so we're going to need to add some logic in Jango to check for this and in order to do that I'm going to go back to the applications forms.py file and in fact before we do this let's go to the jangle admin again and we're going to go back to the products here and if we select the book you can see that the number in stock is still zero and that's because when we tried to place that order for the book that was not accepted by the database so we could not change this value to minus one but the problem here and the reason that we need transactions is if we go back to the homepage and we go to the order model you can see that another order has been placed here so the order exists in the database but that has not updated the number in stock for the associated book product so we now have a database whose stock system is out of sync with the number of orders and this is an example of why we would need a transaction in this case now let's go back to forms.py and we're going to add a function in a second here to check whether we have enough items in stock in order to make a particular order what I'm going to do to start with is Define a custom exception class in Python let's call this exception a product stock exception and that inherits from Python's exception class and that's all we need to do to create a custom exception here and then within the product order form what we're going to do is overr the for the model forms save method so let's write that save method here and that's going to take self and a commit keyword argument which we're going to set to True by default and in the body of this save function we're going to check to see if the product involved in this order has enough items in stock in order to actually make the order successfully so in order to get the order class what we're going to do is call the super classes save method and we're going to pass commit equals false so that it's not going to save the order to the database once we've got that order model back we're going to check if the orders product has a number in stock that is going to allow us to make this order so we're looking at the number in stock property of the product model we're going to check if this is less than the number of items that were ordered in this particular order so if the number in stock is less than the number of items that we've ordered we're going to raise that product stock exception error and I'm going to put this in a new line so we can read the error message we have an F string here not enough items in stock for the product and we're referencing the order here that we have in line 17 and following the foreign key to its product so again if the number in stock is less than the number of items that the US user has ordered we're going to raise this custom product stock exception now otherwise all we need to do is we can check if the commit argument that was passed into the function is equal to true and if it is we can call the orders save method and then at the end we're going to return that instance of the order model so we're overriding the model form save class we're getting the order model and we're checking to see if the order's product has enough items in stock in order to make this order and if it doesn't we raise an exception on the other hand if there are enough items in stock we can then check whether the commit argument is true and we can then save the order to the database if it is true so if we go back to the form that we have on the page here and we try and submit this item for the book this time we get the product stock exception not enough items in stock for the product book and we can handle this exception however we like we can return those errors to the user and display them on the form and this system is working well enough at the moment it will prevent the user from submitting an order if there are not enough items in stock but things can still go wrong on the server we can call this order. saave method and then something could go wrong before we get the chance to update the number of items in stock for the given product remember when we call form. saave if all goes well we're creating an order and saving that to the database but anything could happen in between that happening and calling that update on the number in stock for for example the server could crash and what that would mean is that you have an order in the database for X number of items but we haven't updated the product's number in stock to reflect that order so the database is inconsistent at that point let's solve this by using transactions now let's get back to the Jango documentation on database transactions we've already seen this default Behavior section but if we scroll down a little bit further we get to this section here and you can see that we have a mod in the django.db module for transactions and this transaction module will allow you to combine multiple database changes into an atomic transaction and we can control these transactions using utilities provided by this module now what we have is a function called Atomic and as it says in the documentation atomicity is the defining property of database transactions the atomic function allows us to create a block of code within which the atomicity of the database is guaranted aned and if that block of code successfully completes the changes are committed to the database but if there's an exception the changes are rolled back now there's two ways to use the atomic function in jangle we can use it as a decorator and that can be done on for example ajango view where we decorate the function with the transaction. atomic decorator and we can also use it as a context manager so for example we might not want to decorate the entire view in a transaction we might only want to execute some of the code within a transaction so we use a context manager in Python here using the width statement and then the code that we want to execute inside the transaction is here within the context manager now the code in this function that is outside of this context manager that will execute in Django's default autoc commit mode but what we're going to focus on in this video is using the transaction. atomic function as a context manager in order to wrap a set of actions inside a transaction now what we're going to do is go back to VSS code and within the view we have this form do is valid check and if the form is valid what we're doing is saving the order and then we're updating the order products number and stock we want to perform those two operations as a transaction so let's import the transaction model from D Jango so it's from django.db we're going to import transaction and then back in our view we can Define the context manager here after we've checked that the form is valid we do that by defining a transaction do atomic context manager so we can call the atomic function and then we're going to move some of this code inside the context manager so we want to save and create the order as part of this transaction but we also want to update the products number in stock as part of the transaction as well so I'm going to clean this up a little bit we're going to simulate a server crash in a second so we're saving the order and then we're also updating the number in stock and saving that product details after we do that and these operations are now all going to occur within a transaction so if a server crash does occur after we save the order that is going to be rolled back so that order is not going to be persisted into the database so let's now simulate the server crashing here by importing Python's CIS module and we're going to call the exit function and we'll pass a return status of one there what that means is that nothing below here is going to be executed so what is going to happen to this transaction here we save the order but we're not committing it to the data until the transaction is completed after these lines of code but we're now going to simulate this crash and we're going to see what happens so I'm going to start the Jango server again now on this page we're going to order the video game and let's say we want to order one video game from this system and we're now getting this error coming from the back end but I'm going to go to the Django admin and on here we're going to go to the orders model and you can see there are no orders for the video game that we just submitted so what happened there is that the order was created ated but it wasn't committed to the database and then the server crashed and the database rolled back those changes and that means that we don't have a database that's not in a consistent state that has an order for a product for a particular number of items but that product itself has not had its stock number updated after that crash we've avoided that situation and just to demonstrate that this is a problem I'm going to remove the transaction code here and I'm going to tab that back to the normal part of this function here the indentation and then we're going to try and reexecute this again by going back to our form here let's again try and order the video game we're going to select one of those we get the same error because we're calling cy. exit function if we go back to the Django admin now we have the video game now so the order is in the database we've ordered that video game we've ordered a quantity of one but if we go back to the products and we look at the video game the number in stock was not updated as part of that order so that's why we need a transaction for certain operations in jangle and in databases in general we want to update multiple things but if any part of that workflow fails we want to roll everything back now before we move on I want to highlight in the documentation one other function and this is right at the top here and it's not actually a function it's a setting and this setting is called Atomic requests which is by default set to false but if we set that to True what's going to happen is that every single HTTP request to a jangle view is going to wrap the code within that view and a transaction so if your jangle view is doing any kind of database operations the entire code in that view is going to be wrapped in a transaction now that may or may not be what you're after but you can set this if you want to wrap your jangle views in transactions now let's move on to another concept now we're going to look at how to perform actions when a transaction is successfully committed now sometimes you have to perform actions only when the transaction completes successfully and an example of this might be as we've got in our system here when we order a product after that order has been successfully created in the database we might want to perform subsequent actions after that for example to email the user and inform them about their order but we only want to do that of course if the order has been successfully created in the database so once this transaction that we've got here in this block of code completes successfully we want to then see how to perform an additional action after after that now I'm going to go back to the documentation again and I'm going to go to a section here performing actions after commit and we're going to see that Jango provides a function called on Commit and this on Commit function allows you to register call backs that will be executed after the open transaction is successfully committed and you can see how this is used below here we Define a function that's called send welcome email and that function is passed into to the transaction. on Commit function and what that means is that that when the open transaction is committed to the database it's going to trigger a call back to the function that's provided as an argument so in order to simulate how this works let's go back to vs code and I'm going to go to the top here and just Define a dummy function called email user and that takes in an email address and we're just printing this to the terminal in a real system you could call the user's email user function so we have our function here how do we trigger that function when our transaction here is committed we're going to write the code for that just below this context manager but for now I'm going to remove the code that simulates the server crash and what we can do is we can say transaction do on Commit and then we can pass the name of that function into the on Commit function so the name of the function was email user but that email user function it takes an argument called email what I'm going to do for now is remove this and we're going to see how we can pass arguments to the email user function as well so let's remove that and for now we're just going to say dear user thank you for your order once we have that we can then go back down here and this transaction Doon commit hook will be fired whenever this transaction is successfully committed to the database so let's demonstrate that now by going back to our form here in this page and we're going to try and order a video game here and we're going to select one item from that video game and let's go back to vs code here and we can see we get back this message here dear user thank you for your order and that's because our transaction has been committed successfully it's triggered the email user function and that's printing that out to the terminal now back to the question if we want to provide an email address for example here rather than hardcoding a user how do we do that how do we pass that argument in when we use the transaction. uncommit function well to do that we can use something from Python's Funk tools module and that's the partial function and this is actually documented in the jangle documentation here if you have any arguments you can bend them with the partial function here so what we're going to do is copy the import of partial and we're going to bring that at the top here from Funk tools import the partial function and then when we call transaction. on Commit rather than just passing the email user function in here we can use the partial function to bind that argument so we're going to surround the email user function with the partial call and we can pass the email address in here and I'm just going to use bug bites at test.com so Python's partial function will take another function and it's going to allow you to declare CLE some of the arguments for that function in advance before the function itself is actually triggered so you're partially filling out the arguments and when you call partial and pass in a function a new function is returned that already has this data filled in so let's try this again if we save views.py and go back to our form this time I'm going to select a camera and let's say we want to order one camera if we go back to the terminal you can see this time we get back the email address that was passed in via the partial function and in a real system system of course you would have a user model and you can pass in that user model's actual email address and what happens again if we import the CIS module and call the exit function here that's going to result in the transaction not completing successfully and that means that the transaction Doon commit message or that call back will not be fired and we will not be sending that email to the user so I hope this code and these examples have made sense in order to try and explain transactions in jangle I want to do a follow-up video on this topic where we look at a particular function on Django query sets and that's the select for update function this will lock rows until the end of a transaction and this is a very useful function in certain circumstances I want to dive into that in the next video but that function that particular functionality rather is not supported in SQL light so in order to demonstrate this functionality we're going to have to set up a postgres or other database and I think this video is getting long enough as it is so we're going to move that into another video and I'm going to demonstrate how to use the select for update function in some useful contexts but that's all for this video thank you for watching if you've enjoyed the video please like And subscribe to the channel and we'll see you in the next video

Channel: BugBytes

Views: 4,804

Keywords:

Id: L8k8Ukw1P6U

Length: 34min 14sec (2054 seconds)

Published: Mon Oct 02 2023