Django & Azure Blob Storage - Storing User Uploaded files in Azure Storage

in this video we're going to look at how users can upload files to Azure blob storage in a jangle application now Azure storage gives us a scalable location to upload files of different types and this is a serverless solution we don't need to worry about managing storage and we don't need to think about managing a file system and we're going to see how to use the Jango storages package in order to upload files to Azure storage and also to access already uploaded files from Azure blob storage and we'll also see how this integrates with jango's faile field and finally we'll also see how to generate signed URLs and these are URLs that expire after a given period of time and this allows us to protect our resources from being shared online and that can be important if these are private resources that should not be accessed on the internet so by using signed URLs we can limit the access to our Jango application now if you're enjoying this content give it a thumbs up and subscribe to the channel for more and consider buying a coffee for the channel if you're finding the content useful let's dive in now the package that we're going to use in this video to send files from D Jango to Azure is this one here it's called Django storages and this package provides a collection of custom storage backends for jangle including one for Azure but also backends for Amazon S3 for digital ocean and for Google Cloud Storage so we're going to work with this package in this video what I'm going to do at the moment is open up VSS code and I have a very simple jangle project open at the moment and as you can see we've created a custom user model that inherits from the abstract user now let's say in this application we want our users to be able to upload a file and that should be saved to Azure blob storage and associated with the user that uploaded the file and by using Azure blob storage this has many benefits you can reliably and resiliently store your user data in an Azure blob container without having to worry about the file system or backups and so on so what we're going to do to start with is create a model in this application I'm going to call the model file upload and that's going to inherit from the standard models. model class in jangle now we want this model to be able to link the file that was uploaded to a user so we're going to create a user forign key here so that's going to be an instance of the foreign key field and that's going to be linked to the user and when the user is deleted we want to Cascade that delete and of course we also want to get a reference to the uploaded file so let's create a new field here and that's going to be an instance of the jangle faile field now this field takes an upload to parameter and we're going to send these files to a particular location within our aure blob so let's just call that uploads for now and one last thing we might want to track is when the uploads happen so we'll create a new field called uploaded that and that's going to be a date time field with Auto Now ad set to true now if you're inheriting from abstract user and you have your own user model let's go to settings.py and make sure you have this setting here the off user model should point to that user model class so what I want to do now is go to the terminal and we're going to run python manage.py make migrations and notice that I'm in a python virtual environment that has D Jango installed so let's run that command and that creates a new migration file for our new model called file upload and then once we've done that we can run migrate to make that change to the database and that's going to add a table called file upload to the database now one last thing to do in the setup is go to admin.py and I'm going to paste a bit of code in here we're going to import the file upload model and we're going to register that with the Django admin so now that we've done that let's go to the documentation for Jango storages and we're going to install this so let's copy that command and go to the terminal and I'm going to paste that in to install jangle storages into this virtual environment now once that's installed what we're going to do is go back to settings.py and we're going to define a setting here that's going to tell Jango that the storage back end we want to use is the Azure blob storage now in order to do that let's go to the documentation for Jango storages and we're going to go to the section on Azure storage now as it says here in configuration and settings in jangle 4.2 we changed the way that file storage objects are configured and it's now easier to independently configure storage back ends and also to add additional ones and that's because storages is now configured through a dictionary and this is very similar to the jangle databases setting we have a default storage and you can add as many different storage backends as you like now to send our files to Azure we're going to need this class here from Jango storages and that's the AZ measure storage class so let's just copy this block here and we're going to go back to vs code and I'm going to paste that in at the bottom and I'm going to remove the options for now and in fact let's add one option I'm going to add a timeout and we'll set that to 20 which is actually the default and we can save this file now and if we go back to the documentation there's a section on the different settings we can use with the Azure storage backend and one of them here is the timeout and that is a global connection timeout when you're actually connecting to Azure that's just an example of how we can set one of these settings and we're going to set another one and this is an important one it's called expiration in seconds so let's copy the name of that key and go back to this options dictionary and I'm going to paste that in here and we're going to set this equal to let's say 500 seconds for now so basically when we upload a file to blob storage the URL that's generated when we try and access that file through jangle that URL is only going to be valid for the number of seconds that we put in here and if we set this To None then we must have Public Access enabled in order to look at that file so we're going to see examples of that later on but let's set this to 500 for now now as well as the default storage backend being Azure storage what we're also going to do is add a second storage backend for our static files and we're going to set that equal to just the static files storage and jangle we don't need to change that we're only going to be working with user uploaded files here but make sure you add this because Jango is going to look for that as well so we can now save settings.py but there are a couple of of extra settings that we need to add but these are more sensitive settings and we can get an example of that in the starter code there is an n. example file and it contains an Azure container an Azure account name and Azure account key so we need to set the three of these based on what we actually add to Azure the Azure container is basically going to be the container for storing our blobs and then we have a storage account name in Azure and the Azure account key is going to allow us to connect securely using a key that's specific to our storage account now we'll fill these values in in the next section so let's now move on and go to the Azure portal I'm going to go back to the browser and I'm going to go to this portal here so the first thing I want to do is create what's called a resource Group and you can see that here and a resource Group is essentially just a logical container for a group of services that are related in some way so you can imagine for a jangle application you might create a resource Group in order to put the server where the application is deployed and also to put the Azure blob store storage account where your files are going to be stored and maybe you'll have a database and so on so a resource grrip allows you to logically bundle those together and imagine you're going to delete that entire service if you bundle them together in a resource Group it becomes easy to just remove everything associated with that group so I would recommend in Azure to just add things to the specific Resource Group that they belong so we're going to create one of these now in Azure so let's create a resource Group and we get this form here and what I'm going to do is give this a name of Jango Das web app now you can select the region that's closest to you I'm going to go to UK West and then we can review and create this we don't need to review it we just need to create this so let's hit the create button at the bottom and at the top right you can see the resource Group has been created so now that that's done what we can do is we can create a storage account that's part of this Resource Group so let's go to the search bar at the top and one of the recent services for me is storage accounts if you don't see that you can search for it in the search bar and that takes us to the storage accounts page so all access to Azure storage is going to take place through a storage account so if you're using blob storage or Azure tables or Azure cues these are all different subservices but they all fall under the domain of a storage account so we're going to create a storage account now using the create button that's going to take us to this form so one of the things here is the resource Group make sure that you add that to the resource Group that we just created called D Jango web app and you need to give the storage account a name and this must be unique across All Storage account names in Microsoft Azure so I'm going to paste one in here called bug bites Django storage and for performance you can use the standard one unless you have requirements for a premium version and you also have some redundancy options so Geo redundant storage that's slightly more expensive but it backs up your data in different regions whereas locally redundant storage that's the recommended type for non-critical scenarios I'm going to select that that's all we need in this video and then then once we've done that we can create our storage account by clicking the button at the bottom that's going to validate that and then after validation we can hit the create Button as you can see at the top right that deploys this storage account so that's in progress at the moment what we're going to do is we're going to copy the name of the storage account and I'm going to go back to VSS code and within this project we have the example.in file I'm going to create uh n file from that so let's go to the terminal here and I'm going to clear the terminal out and what we're going to do is copy n. example to a n file and that's going to copy the keys from the example file into this and then we can set the properties in this so the Azure account name that relates to the name of the storage account we're going to set that to bug bites Jango storage and we'll go back to the other two in a second let's go to the portal and we can see that the deployment of this storage account is complete we can now go to this resource and we have that open here and this is the container for our storage account if we go to the left hand sidebar I'm going to go down to the security and networking Tab and we can go to the section on access keys and what I'm going to do here is copy one of these two keys that are provided so let's do the first one at the top here I'm going to copy this key and we're going to go back to the vs code project and for the Azure account key I'm going to paste that key into that variable and we can now create the Azure blob container let's go back to the portal and I'm going to scroll up here and it's this section on data storage now aure storage accounts you can create multiple different storage types the one we're interested for our jangle application is containers so let's click that and we're going to add a new container using this button at the top left and we'll give that a very simple name called jangle container and notice Anonymous access level here this setting specifies whether data in the container might be accessed anonymously and by default this container data is private to the account owner now at the moment we can't change this setting because Anonymous access is disabled on the storage account we'll see an example of how to change that at the end of the video but for now let's create our new container and as you can see that's called Django Das container so I'm going to go back to vs code and we have one more setting here so I'm going to set Azure container to that Jango container value so these are secure variable names and particularly the account key must not be committed to GitHub what we're now going to do is we're going to read these into our Jango settings and we're going to use Jango Environ to do that so let's go to settings. Pi and I'm going to scroll to the top here and what we need to do is install Jango Environ so to start with I'm going to go to the documentation and I'll leave a link to the documentation for Jango Environ below the video this is a package that allows you to use the 12 Factor methodology to configure your Jango app using environment variables so let's go to the quick start section on the left hand side and we're going to copy some of these values here so let's start by importing the package at the top of our Jango settings.py file and and then we're going to paste a couple of lines of code in here we're going to set up an environment object so let's go back here and just underneath the base directory we'll do that and then the only other thing we need to do in settings. pi is read the values from the environment file by pointing jangle Environ to that file so I'm going to copy this and we're going to paste that into our settings now we haven't imported the OS module but we have a base directory and that's a path li. paath object in Python and we can reference a property or rather we can reference a file in that directory using this syntax here so what we're doing here is we import Environ and we set up an environment object and then we use the Environ reden function to readen the values from that environment file now once we've done that I'm going to go to the bottom of this file and I'm going to paste in these lines of code here so we're setting the settings here for Azure container account name and account Key by reading them in from the environment using that inv object so that's going to get these from this do n file and read them into the settings and that's what we need in order to actually connect to that storage container and to use this Azure storage class from Jango storages so let's now move on on the terminal I'm just going to make sure everything is up and running and by the way if you haven't already done so make sure you have installed Jango Environ so we can use pip install Jango Environ to do that and I already have it installed in the virtual environment let's clear the terminal and now we're going to actually allow a user to upload a file and and save that data to this file upload model so we're going to create a jangle form that allows the user to upload a file and that's going to be a model form and it's going to be centered around the file upload model so let's create a class here called file upload form within forms. pi and that's going to inherit from the forms. model form class and in that class we Define a meta class and the model we're linking it to is file upload and we only want one field on that model and that's the file field so if we look at models.py we're going to dynamically link the upload to a user within the jangle view the file itself is what we are going to upload so we need that field in the form and then this is automatically set by the database when we save the record so we don't need to do anything with that we only need one field in this form now we need to actually render the form on a page so let's go to views.py and we have a very simple jangle view here now because we want to associate the upload with a user we're going to protect the view with the linore required decorator and we need to import that at the top so I'm going to do that at the top here from Jango doc. off. decorator now what we're going to do if it's a get request is just add a context dictionary here with a key called form and we're going to instantiate the form that we created here and that's the file upload form so what I'm going to do is import that form into the views.py file so from forms let's import the file upload form and we can instantiate that here in this key and that's going to add it to the context and it's going to make it available in index.html so we're going to go to index.html that's a template in the c/ templates directory and if we go in here we're going to remove this H1 tag and for now I'm just going to render the form out and we're going to do that using divs now let's test this out for now we're going to run the Jango development server and we're going to go to the browser and we're going to navigate to that page now we're getting a 404 not found and that's because we're not logged in and in order to log in what I've done is create a jangle super user and that has the username of admin we can log into the jangle admin with this super user and then we can navigate back to the main page of the application and you can see that field for uploading the file so that's not very useful at the moment we don't really have a way of submitting this data so we're going to add some code to that in a second let's go back to VSS code and we're going to minimize the terminal and I'm going to paste some different code in here so let me do that now we're creating a div with the class of container and now we have a form element in HTML and the submission of the form is going to be a post request and importantly we're adding the ink type and that stands for encoding type and we're setting that to multiart SL form data now this attribute is very important if you're uploading a file through an HTML form make sure to set the encoding type using this attribute now because the submission is a post request also make sure to add the csrf token and then we've moved the form. ASD call into this form element and finally we have a button that allows us to submit the form so let's go back to the page and refresh and we now have the submit button everything else is the same and we're not going to change the Styles here because this is not about the CSS obviously you wouldn't deploy a website that looks like that but for now what's going to happen is that when we submit the form nothing is going to happen because if we go back to views.py this is only going to reinstantiate a context and return the same template with that context to the client now what we actually need to do here is handle the the case when the request. method is a post request so let's do that now if request. method is post we're going to create the form and in order to create the form we instantiate it as we did before but this time we're going to pass some extra data into the form that's coming from the client when it's submitted so request. poost will be the first argument and then also request. files we need to pass request. files because we know that a file is going to be submitted in this form so poost and do files these are two attributes on the jangle request object and they capture the submitted data so we pass them into the file upload form once we've done that when we have the form we can check if the form is valid using form do is valid and that's a function if it's valid we can then process the form so I'm going to create a variable called upload and we're going to call form Dove and pass commit equals false now model forms when you call do save it's going to try and save that to the database unless you pass commit equals f and that's what we're going to do here because when we save the file upload if we go back to models.py we don't want to save this with just the file because we want to also associate a user with the file upload so let's do that now after we've got the object back here we can set the user on that so upload. user is going to be equal to request. user and we know we've got a logged in user because we're using the login required decorator so once we've Associated the user with the upload we can then call the do save method to save that to the database and finally just for Simplicity I'm going to return a redirect back to the homepage now we need to import redirect at the top so we can do that from jangle do shortcuts now what if the form is not valid we can add an else Clause here and handle that and what we're going to put in here is very similar to the get request case so I'm going to paste these two lines of code and what we're actually going to change though is what the form key is set to so rather than instantiating a new form we can pass in the form that has the Errors By referencing that form here so remember form do is valid return false that means there's errors and we're going to hit this else block and we want to add that form to the context so that we can see the form with all the errors so let's try this out and the important thing from the perspective of azure is that when we save the upload the uploaded file data is going to be present in the model in the file field here and that's going to be sent to Azure blob storage because in settings.py we've set the default storage backend to Azure storage and on top of that we've read in the connection information and the blob storage or the container information from the environment using these environment variables let's now test it out we're going to go back to the browser and I'm going to refresh this page and I'm going to choose a file that's in my local directory so I have a cities. CSV file here and this just contains some information about cities in the world the information itself is not important let's try submitting this and when we submit that you can see the request is in flight and after we submit that we're redirected back to the page and that you can see that here in the view we're redirecting back so what we're going to do now is check that the file has been uploaded to Azure to that container and blob storage so let's do that now we're going to go to the portal and we're going to go back to Microsoft azure's storage account so this is the storage account we configured earlier on we're going to go to the containers and we're going to find the jangle container that we created and notice now we have a directory here called uploads and if we look in that we get a file called cities. CSV so this process has now worked we have a form that we created with jangle and that form is being rendered in a template and when we upload the form on this page it's sending that form to our Azure container and storing it securely and reliably on the cloud I want to go to the Django admin UI so let's go to/ admin and we registered the file upload model with the admin so let's go and have a look at this now and you can see here it's associated with the admin user that I'm logged in as and we also have the file below now this linking of the user to the file that's something you're going to commonly want to do if you're allowing users to upload files to different locations you want to know which user actually owns the file or which user has uploaded the file so you can create that link in your models by adding that kind of foreign key between the two entities now one thing to note if we click the link here in the jangle admin to the file that's going to actually download the file I don't know if you've seen that at the top right here we have a download history and it contains the file that we have uploaded to Azure blob storage so through the admin UI or through the templates that you have in your application you can reference these files that are stored on blob storage and you can then show them to users or allow users to download them now what I want to do to finish the video is go back to this page here and underneath the form I want to show a list of all of the logged in users files that they've uploaded so let's go back to the Django page here and what we're going to do is just keep this simple and add this to the context on a get request so I'm going to split this into a couple of lines here so as well as the form we're going to add another key to the context called files and in order to do this we need to get an access to the file upload model so let's import that at the top and then go back down to the context dictionary we're going to paste the name of the model and we're going to perform our function here and it's file upload. objects. filter and we want to get the uploads that are associated with the logged in user so we're going to filter on the user field and we're going to get back those files that belong to the user and we can get the loged in user on the request. user property once we've got the files in the context dictionary we can then reference them in index.html so just below this form here I'm going to create another div and I'm going to give that an ID of files and we can close that off at the bottom and what we're then going to do is we're going to create a template for Loop here and for each file in the list of files what we're going to do is we're going to render them out using an anchor tag so let's create the anchor tag and it's the hre property that we're interested in here we're going to refer to the file and the name of the field that was on that model was also file and it was the URL of the object that we want to get back here and display in the h r and I'm just going to give some hardcoded text here of view so just to explain this file. file each one of these files that's coming into the template is an instance of this model class here and for each model there is a field on the model called fail so that explains why we're referencing that here and a file field in jangle has a property called URL so if you need to get a link to that file you can do it through the URL property so let's save this and go back to the page when I refresh this page you can see we have a single link here and that's because we only uploaded a single file at the moment to Azure blob storage if I click view again that's going to download the file at the top right so this gives your users access to the files that have been uploaded to blob storage now let me quickly upload another file and I'm actually just going to upload the do get ignore file for this project let's submit that and you can see now we have a second link appearing on the page so there are two files now associated with the logged in user and they are being displayed on the page albeit in one of the worst uis youve probably ever seen now let's finish the video with a couple of different concepts here I'm going to go back to settings.py and to the storage setting and we set the expiration seconds here to 500 seconds what I'm actually going to do is comment that out and let's assume that we are not setting this variable in this setting if we go back to the Django storage's documentation let's look at the expiration seconds value what this means is the number of seconds before a URL expires and if we set that to none that means that the URL is never going to expire and notice that the default value is none so when I comment this out that means it's going to go to none by default and as it says here that means it's never going to expire now this is important be aware that the container must have public read permissions in order to access a URL that does not have an expiry date now these kind of configurations can trip you up so I'm going to explain this and we're going to walk through the example of what this means now that we've commented out expiration seconds that means that our URLs are never going to expire so let's go back to the application and I'm going to refresh the page and let's click one of these links and this time we get a different message we don't get the file being downloaded instead we have a public access not permitted exception so basically what this is telling me is that Public Access is not permitted on this Azure storage account and that conforms with what the documentation said a container must have public read permissions in order to access a url url that does not have an expiration date so let's go back to Azure and if you need your objects or your files to have public access and you don't care about expiring URLs then you can set that on the storage account and on the container so let's start with the storage account what we're going to do is go back to this page here this is our storage account it's called bug bites Jango storage and we're going to go to the Overview at the top and click that now in order to turn on public access what we can do is go down to these properties and there's a section here or there's a variable called blob Anonymous access and currently that's disabled now I'm going to click on that and if you need to enable Public Access what you can do is is you can change that from disabled to enabled once you've done that click save and that's going to apply that new setting and you can see that that's successfully updated so now that we've turned on public access on the storage account that means that we can set this public access on the different types of data storage that we're using so let's go back to containers and we're going to go to the jangle container that we were working with and there's a section here called change access level so if you need to change the access level of a particular container you can do it here here now you can see that this is grayed out at the moment and it says that Anonymous access to the container is being blocked because it's disabled on the storage account so I think it takes a little bit of time for that change we just made to be applied so what I'm going to do is just refresh this page and hopefully that setting or that change is going to be applied when we hit change access level again and you can see now that is no longer grade out so we're going to change this from private with no Anonymous access to the blob setting here and this is going to allow anonymous read access for blobs only let's do that now and set that to okay and that successfully changes the access level for the containers we can now go back to our jangle application and that was here let's go back to this page and this time when we hit view again it's going to download the file and that's because Public Access is now permitted so if you need that Public Access you can do that on your Azure storage account what I'm going to do is change this back though because that's generally not what we want in ajango application if you want to secure user uploaded data and this is all contextual depends on what you're actually developing and what kind of application you have what I'm going to do is change it back to private only so we've done that on the container let's go back to the storage account now and on the overview page we can go back down to this Anonymous access setting and we're going to set that to disabled again and save this configuration so for security unless you have a good reason to turn Public Access on you should keep your objects in your blob storage containers private now we can finish by looking at expiration seconds in a bit more detail so let's set that to a value that's very small let's say 10 seconds until the URL expires if we save that value and go back to our jangle application and refresh this page that means it's going to generate some URLs for these objects and I can click view here and it's going to download that file as you can see at the top right but after 10 seconds that URL that is generated for this object that is going to expire and that means we're going to get an error when we try and download this file after that expiration time so let's click view now and you can see again we're getting authentication failed this time and the server has failed to authenticate the request and if we look at the details of this error we can see that the signed expiry time must be after the signed start time so the URL for this object in Blob storage has now expired and that means that you can protect the URLs that you're sharing in your applications so rather than having a URL that never changes if you set this expiration time and keep your objects private what that means is that you can generate these URLs and you can share them with people internally on the application but if someone copies that and pastes it all over the Internet that's not going to compromise the data in Azure blob storage because these people might be sharing the URL but it's going to expire after a given time period and as soon as it expires the link that they've been pasting all over the Internet becomes redundant becomes irrelevant and all is well in your application now depending on what you're developing you can set this to whatever whatever value you want 10 seconds is probably too small a value I would say for almost all applications but you can set that to something like let's say a th000 seconds and that's going to very quickly expire in the grand scheme of things and it's going to prevent your objects being shared across the internet so that's all for this video we've seen how to use jangle storages with Microsoft Azure as a storage backend and we've also seen how this expiration seconds value Works in order to configure URLs that expire after a certain period of time and we also saw how seamlessly this Jango storage's back end works with jango's file fields and we saw how when we submit a form we can upload the files to Azure blob storage and also how we can retrieve files from storage and download them within our applications so thanks for watching if you've enjoyed the video give it a thumbs up and subscribe to the channel for much more Azure and jangle material and if this content was useful consider buying the channel a coffee we've got a link in the description thanks again for watching and we'll see you in the next video

Channel: BugBytes

Views: 1,466

Keywords:

Id: mfYi_nnn69E

Length: 31min 3sec (1863 seconds)

Published: Mon May 20 2024