Learn Django - Implementing AWS S3 Storage for Static and Media files in Django

hello and welcome back to another tutorial so in this one we're going to be looking at static and media folders so i'm going to give you an overview on how to set up static and media inside of your django application and then we're going to configure aws s3 the simple storage solution from amazon and we're going to have our static and media files be inserted from the aws service so we create a small app an upload image application we'll do that very quickly you can skip over that if you want to and then head into the aws setup but i will go through an overview of how to use and implement static and media folders inside of your django application before then moving across to configuring aws to serve those files from the s3 service so in addition to that we'll have a look at or at least our brief view on acls their access control lists and also setting up policies based upon groups and users so by the end of this tutorial you will have a baseline of knowledge of how to set up your django application with s3 but you ought to go then and move that forward and just do some more reading and before you start implementing this into a production environment you'll want to really grasp the idea of the securities that are in place to protect your files because once you start allowing users to upload files to a server then you've got potential security problems this tutorial is split into two parts in the first part of the tutorial i'm just going to go through for those who need need it how to set up the media and static files within django so i'll create a small project very quickly and we'll set up the media and static folder for those files so once we've done that we then migrate our media and static files over to aws so if you're already comfortable with media and static files and you just want the aws part then just use the timeline in the description and just skip ahead to the aws start so here in django we have two repositories we have the static files or static folder and the media folder so the difference being that the static folder typically we're going to use that for files that our project is dependent upon so javascript files css files other code files that our website potentially requires so having that folder separate from the media folder means that we can apply different security privileges and manage it differently from the media folder so the media folder tends to be a folder or an area which we can allow users to upload images potentially or where media for our project will reside so this typically has different permissions because we want to allow users to actually upload and potentially interact with that space so first of all we're going to set up the static folder and the media folder here in this development environment so i've gone ahead and built this simple django project the project's called core and i've made a new application called demo so first of all we're just going to set up a very simple file upload system and then handle those files that are uploaded using the media folder so because we're using media we're going to pip install pillow and then we're going to need somewhere to actually store the media folder or sorry store the media files so i'll create a new folder i'm just going to call that media of course you can change these folders to absolutely anything you want in reality i'm just going to call this one media so i'm just going to build a very simple interface in my demo app which is going to allow users to actually upload files so we can simulate the uploading of files and then storing within the media folder so that we can see the media folder in action we've gone ahead into the demo app in the models and we've just created a very simple model here so just to give you some detail on this if you look at the post table you can see that we have the image image field so we're going to upload an image and then just define the upload path so here we're using this new method or connecting the upload to any method called user directory path we take in the instance and the file name so the instance allows us then to utilize other items so for example here we're taking in the instance id which doesn't actually work i just wanted to point out that we could actually for example here type in for example user id so we could capture the user id and utilize that as part of the actual link to where the image is going to be saved so here what we're trying to do is to format the actual path of the image so inside of media we're going to create a new folder called posts and then we potentially want to use some sort of identification so here for example like i said we could potentially use the user id so that we can then identify that folder to a user and then the last item here we have the file name so this link ends up being media slash post slash the user id and then the file name you can alternatively just select or define a default path right here if you wanted to so i've gone ahead and created a form here this is just going to be the form that we're going to utilize to allow users to upload the images so we're just taking in the title the image caption and the image which is going to be the actual image upload facility here in this form i've added a few different items here so these are just bootstrap items so then we head over to the view so we're going to take in the upload form and we're going to use a form view here so we're taking the template which we're just about to build obviously the form class is upload that's the form and then we have the success url which is just going to be the same page so it's just going to link you to the same page so we validate the form we save the form and then we print the form clean data and that's just going to print to the console if you're not familiar with printing and that's about it really in the view so let's go ahead and just configure our urls so here we have the upload so here we're just linking to our upload view remember remember we're using class based views here so as view and then gives the name upload the app name is demo so that's in our demo application obviously now in the core i need to link to that url so let's just get rid of all that so i add the link to the demo urls and then i just need to also include right there so that's the urls in place and then just finally we need to add the template so go ahead in the demo i'm going to add a new folder called templates and then inside of here i'm just going to drop in a a new file called index so on this page we've just created some bootstrap code here and i've just added the form so the form is right here and we're just adding the form here so i am using a base here so in addition to that i've also just built a very simple base so this is just a bootstrap implementation here to change the title so you can see that we are extending from the base okay so with that in place let's just check the server and refresh our page and you can see this is what we have a very simple upload form so we just need to type in the title and maybe the name of the caption and then choose a file and then upload so this is an example of using media folder so now we need to actually configure the media folder so let's remember that we're working in a development environment here the configuration for a production server would be slightly different so just to note that we are using a development environment so to get the media started or working go over to the core and in the settings at the bottom here we just need to add some additional configuration so what we're trying to do here is just tell django where all the resources are so in our settings here we've got two settings that we can configure for our media route and these are the two settings here so we've got the media url and then also the media route okay so let's go ahead and now we've configured that there's an additional config configuration that we need to do so we are using os because we're trying to create a media route with a path the base directory of our application and the folder media so let's go up here and also just import os i'm going to need that so those are the two points that we need to create here in the settings file so here we have two settings just to cover this quickly then so we have the media root which is a an absolute path so the os path the base directories and the media so that's going to create an absolute path to our media folder and then we have the media url because obviously when we serve our web page it's not going to be utilizing the absolute path for our media route it's going to be potentially utilizing the url so here we have two settings that we configure so now we have that in place for a development server we have an additional configuration if you go over to the urls we're going to need to add into here an additional configuration so let's go ahead and import so we're going to need to import settings and we'll just set this up for static as well um so we only have to come here once so we can do a bit of that and then what we need to do is then just define so i've created this bit of code here now we just defined this is obviously just for development only so if settings.debug so settings being the settings file and the configuration debug being true so if the debug is true you can see here that we've set up some url patterns for the static url and the media url so here we're just trying to simulate a production server url essentially that we can utilize with within our own development environment so that needs to be in place so you can see that i've done it for the static files which we'll cover shortly and the media folder if you wanted to read these just need to notice that again we're using settings media url so we import settings here and we want to utilize media url which is obviously a reference point in the settings to this media url right here okay so now we have that in place let's go ahead and just make sure this server's on okay so i've gone ahead and chose a file and included title so press upload you can remember that in the view i've just made it so it loops back to this page so it looks like it's worked so let's go back into the code here so what we should have now in the media is a a photo here called post and then none so the reason why it says none is like i was telling you earlier that this is just for just to give you an idea of how you can utilize this instance here because there isn't an actual id in this post then it's going to return none so you can see that i've uploaded some files here so every time it's uploaded it's utilized the file name and notice that when you upload a folder or sorry a file that with a similar name django automate automatically puts a random string on the end of the file name so you do end up with files or similar files but with just different names so it might be that you would configure to check to make sure that if an image exists it overwrites depending obviously how you want this set up so there's a quick overview on how to set up the media folder so now let's think about these static files so the same type of process really here i'm just going to create a new folder called static for this example i've just created a simple static css file here just with some simple css so let's have a look at how we can actually connect our static files to the templates so if we go over to the templates in demo in the index here or potentially the base so what we want to do is we want to import our css into our project so you can see here that first of all what we need to do when we want to include static files is to actually load them in so load static so that needs to be on each page that we want static files and then you can see here we reference static in this href to link to this css style sheet and then we just define the actual file we need and obviously this is a file path right here so if you had a folder structure inside of your static folder you could apply that right there so to get this static files to work we need to go back into now the core and at the bottom here similar to what we've done with the media we also need to now set up the configuration for our static files so in windows we have three potential settings here static root static url and the static files directory so this is obviously the absolute path for the static files we have the url and additionally the root so the static route this is going to be helpful and utilized for when we utilize the collect static command before we place our application into a production environment we would typically run collect static command so django looks for all the static files in our apps and it collects them all and then puts them into the static route so let's go ahead and just run the static collect static command so collect static you'll see that there will be a problem because we haven't actually defined our static route yet so let's just go ahead and define static route and then try that again and now you can see that we have 133 static files copied to static now notice here where it's actually stored the c drive uh static so that isn't obviously what we want here so you can see what's happening there so we configure this later let's move on to the next step so now we've configured the static here we should now go back to our web page and we can refresh and you can now see that it's picked up our css file and is actually utilizing our static files so that was a brief introduction to using the media folder and static files within django let's remember this is just a development environment so as i keep saying you would configure this differently if you're using a production server so now that we have an example application that we've just built let's now go ahead and configure our application to serve the media and static folder from aws so go ahead and sign up and i'll see you inside of aws so just a quick note for those who haven't used aws before you notice that when you log in you've got two options a root user and i'm user so here we have the potential capacity to create additional users um but also think of this as a the application as a user so we can define very specific control and security over our applications based upon what resources we want them to access so we'll have a look at that later in this tutorial but you also use the root user to begin with if you've just signed up to login okay so welcome to aws the management console so from the management console you can't see this online because i've just hidden my username but if you go over to services you can then type in s3 this should bring up the scalable storage in the cloud so we're going to use the sre service to store our media and static files and have amazon or the aws service and serve our media and static files to our django project and this can be done in a development environment so let's go ahead and set up s3 so we can see here that we are going to create a new bucket so let's go ahead and create a new bucket so i'm just going to call my bucket django demo and you can select your region and so on and then from this example we want to just deselect blocked or public access okay and then we don't need anything else we're just going to set up a a basic baseline configuration and you can then go and explore the different options already exists okay please acknowledge that okay so now we have this new storage point where we can now potentially access our media and static folder so aws provides a service called im so it allows us to set up other users essentially which we can then assign to individual projects so by doing this we can micromanage these users based upon the access that they're required based upon the applications that they're associated to so instead of having one admin user which we then utilize to access our applications or access aws through our applications we're going to set up individual users and we can then apply individual security to those users which can then be utilized individually for individual applications thus providing better security to our aws service so if you go over to um services apologies if that didn't make sense i kind of embellished there a little bit um let's just type in i am so we've got the management services so essentially we're just going to make a new user which would then going to utilize within our application to connect to our s3 bucket so here you can see that i've already got a user in a group so let's just go ahead and create a new group i'm going to call this s3 demo group so let's just create the new group sorry s3 demo got ahead of myself then so next step and then we're going to give this a policy the aws s3 so we're going to give them a policy this is a policy so this is a security control script so here is going to define what access this group can have to the in this case amazon s3 so here is a default access policy called amazon s3 for access so i want to give the users full access because i need to upload files and i need to download files so users are going to need to be able to upload their files to s3 bucket our amazon bucket here so we're going to need full access to begin with so let's just press next and then create group okay so now we've got this group here called s3 demo we can now make a user and apply it to that group so let's just go ahead and create a new user so i'm going to call this user s3 demo so we're going to need a programmic access so this is a setting that we want because we're essentially needing a an access id key we're going to need to use that to configure our django application to access our aws service so i select the promatic access programmatic access and press next and then i want to add them to a group so the s3 demo group remember that has amazon s3 full access to our folders we created earlier so press next and here we're just going to press next okay so this is the setup we've made a group we've now made a user and put them in that group remember that the group has a policy set so the security policy to s3 not the user okay so we create user and there we go so now we have an access key id and a secret key so you can go ahead and download that and i will utilize that shortly so that's going to provide us access to that folder so at this point i just want to note that this demonstration is just for demonstration purposes only and for you to get a grasp and an understanding about some of the aws technologies and settings that you could apply in your application this isn't by no means a a robust production setting i'm assuming that you're new to aws you've never used aws before i'm just introducing you to the service and some of the features that it offers so now that we've configured the aws service or at least allowed access we should be able to now access our bucket we now need to configure django with a few different packages and then a little bit of configuration so let's go ahead now and actually install the two packages that we're going to need for this so the first package that you're going to need to install is botox 3 which is the aws sdk for python so is the amazon web service software development kit this is going to allow us the services and tools to actually create the connection and upload etc and then just to make the process easy for us to implement we're also going to be installing django storages so like it says there's a collection of custom storage back ends for django so we're just building upon boto just to make things a lot easier for us to configure without having to actually create our own packages to actually utilize the s3 storage in any way this should make the process a whole lot easier so i'll go ahead and pip install the photo 3 tools and then we will also install django storages so go ahead and pip install install django storages okay so now we have all the packages that we're going to need we can now head back into the core and our settings this is what we need to do next okay so i've gone ahead and now configured the settings needed to have let's have a quick look at these so you can see that we've got aws access key id which is the id key id that i downloaded remember when we created the access we we had some use user new user credentials and i downloaded the file for that so the new user file so this is what it looked like we had the username the access key and then the secret key and then the console login link so i use these details here to um configure the access key and then the secret access key so my storage bucket was called django demo1234 so we can just double check that so that was the name of my bucket and finally we then go down here so this is just some setup configurations and we have the aws default acl so i'm going to set this to public read so amazon provides us with some default acl so access control lists so here's defined as public read now i think by default if we don't include this setting the default acl will be private which will mean that when you upload the files they will only be accessible for a short amount of time and then you won't be able to access them through your system so public read is going to provide us essentially a method of being able to read the files but not actually potentially write to those files or folder so that gives you an idea of why we're utilizing acl so if we head over to a page here this is going to be well worth a read on this page the access control list overview have a look through here and get an idea so we're using the canned acls so there's a number of different options here just have a read through and this is where it starts to become a little bit overbearing potentially if you're new developers just wanting to just wanted to get a website out there aws can be a little bit of a a monster to begin with until you start learning a little bit more about it now it is always recommended i always recommend for new developers too before you start using aws and storage online because it is quite dangerous to have a folder open for users to be able to upload to so you kind of need to know what you're doing before you set out this in a production environment so potentially utilizing hosting first which hosting before you start utilizing these type of systems may be more recommended but it's definitely worth reading through like i said this this and many more pages of different settings that you can apply okay so let's get back to this configuration so you can see that we set to public read so you can read about what that actually applies um but it is as it like it says we allowed to allows the public to just read and not write potentially so we then have the aws location so we're going to set this up for static to begin with and then we define the static file directory which is going to be the os path join of the base directory and the static so again that's just our absolute path to the static folder that we created earlier which was somewhere around in our project okay just here okay so nothing ever goes smooth so i've gone ahead and used the configuration now just to start off with i've configured the aws default acl as none so i did show you read only so i'm just going to start off with none so we can start from a baseline here and then i've gone ahead and in the control you can see i've typed in manage pi collect static so django is now going to collect all the static where it now would normally put it into the static folder in our project it now has uploaded to our aws service now notice that there's a hundred and thirty three stack it static files copied so what that represents is a lot of the background css and javascript needed for the django admin area so that's what those static files typically represent so if we now go back into our aws project you notice that i made a new bucket in actual fact here um i had to stop and do something and i've come back at a different time and i created a new bucket so it was the same process like i've shown you now if you do have a problem whereby you go through the process and it says access is denied you don't have access to put and to upload files and just go ahead and just delete the bucket and just go through that process again that's all that you need to do i promise that's the case so you can see we've uploaded to our bucket here the static folder inside of here we have the core like we did in our project and like i said the admin folder this is going to have all the css fonts image and javascript for your admin area when you type in the url admin okay so now if you head over to the small app that we built we should now be able to refresh once we've start this server of course and then we should see that the the amazon is actually providing us our file so let's go to page view page source you can see here that we're actually being served the css page right here you can see straight away though that we have access denied so this is a problem that might occur and notice that i used the aws default access none so just reading through the um documentation i think there's been a few changes in how these packages serve the file so if you want to first of all uh complete it with the acl as the default then this is the problem that you get it's going to occur i believe it utilizes the private feature or the private acl which is going to make the file available shortly but then will make the file private so the files do get uploaded but you won't be able to access them so you'll need to go through this process if you go to the management uh your bucket so just uh delete your files go ahead and go to the actions and delete and then just run through the process again because you'll notice if you use the public read acl that will then apply the public reading acl to the files and then if you go ahead and like i've just done here and collect stats again and just have those files copied across you'll then be able to go into your project and as you can see it's now being served so if we go back into the view page source and head over to the link you can see we can now actually access the files so you can see here the key to accessing or the key to protecting your files is going to be reading those access control lists that i linked to previously in the tutorial at least that's going to be one step of a successful secure environment so here to generalize we have two different securities we have folder and file security and we have the actual user that obviously the user we created earlier which is utilized to create an api to actually connect our application to our service now if you remember if you go back into the i am service we created a user and we gave it we inserted it or included into the group and that group had policy that had a group had a policy of the s3 policy which was the full access now we don't necessarily want our application to have full access to our buckets so that might mean creating new buckets etc we know that django should only have access to a very specific bucket that we created and potentially then specific folders within that bucket in this case they should only have access or the application should only have access to the static folder and the media folder that we're going to create so let me just give you an example of how to make a custom policy for our user so inside of the i am service here we have policies and we gives us the opportunity to create a new policy you can see that i've already created one so go ahead and select that and then you can just select the json so if you know what you're doing you can just apply this but if you explore the visual editor that gives you more of a gui interface and you can look at all the different options that are available to create your different policies so here's an example of something you might want to set up so we know that we only want this program or this user to or the group to access certain folders like i said so here we've set up an action so it allows them to access the s3 services but then we define what resources they have access to so they have access to the root this is the actual bucket name in my case i've changed the bucket name so this is my bucket that i want this user or group to access only and obviously i want them to be able to access all the other folders and files in there so this is a simple overview an example of how you might want to then further apply security to the user or group to only access certain resources in the s3 service so we can review the policy here and then we can give it a name we'll call that s3 django demo and there we go so let's just uh create that policy now of course we want to apply that policy to a a group in this case so let's go over to the groups and then we've got permissions here and you can see that we've got this attached policy so let's just um this is interesting the simulate policy you can actually check policies against different rules etc that'd be worth exploring but we can go ahead and attach a policy so we called that sri django demo now i think it was this one i created so attach that one so we can now obviously detach this policy here so they only have access now this user or the users in this group should only have access now to those folders so that should now be in place so let's just test that out in our application and it still seems to be working so it looks like everything is still okay now you could obviously go ahead and if you made a new bucket you could try and utilize this user to access that bucket and of course that shouldn't work so that would be a way of testing this policy but that just gives you an overview of policies that you can apply to a user to further enhance the security in your application or the s3 environment so next up we want to obviously move ahead slightly and also apply the same settings for our media so we're now going to set up a custom folder to utilize for the media so we're going to set up some custom settings so in the core i'm going to create a new file i'm going to call this storages so we're going to set up a a custom storage for our media so you can see here i've imported the s3 botox 3 storage and then created a new class here called media store and then i've set some parameters here so the location which is going to be a folder in the s3 bucket called media and i've got file overwrite of false so this is just an additional option that you might want to choose for this type of media so do you want the users to be able to overwrite other people's files for example if they had the same file name so that's just something for you to think about so now you can head back into the settings of your core and here we set the default file storage and that's going to be core storage's media store so the course storages media store so that's what we've just set up okay so if you go back into the application so here i've uploaded or i've selected an image called django png i give it a title i press upload of course not forgetting to start the server first and then press upload so hopefully what's happened now if we head over back to our aws s3 bucket we should have a new folder called media and the file that i've just uploaded was going to be in post remember after post is none because i haven't configured that properly and then we have the folder or sorry the file that i've just uploaded so you can see that our media folder is now also going to be served from aws okay so there we have it that was a short introduction to aws or utilizing django with static and media folders and also utilizing the aws service to serve those files to your project so probably like all tutorials here on this channel there probably ought to be a follow-up to this maybe we can explore some of the more specifics of setting up a production environment for these folders um that's something that we could look at or we could explore a little bit more about policies and access control lists etc i'm happy to take suggestions on what you would like to know and to learn hopefully there was something here that you've learned and you can now go and apply that in your application or at least test in your development environments

Channel: Very Academy

Views: 8,020

Rating: 4.8672199 out of 5

Keywords: django tut, django tutorial, django 3, djangoproject, django examples, learn django, django beginners, beginners django, django framework, django 2020, django example, django ajax, django aws, django static, django media, django s3

Id: ahBG_iLbJPM

Length: 40min 40sec (2440 seconds)

Published: Mon Aug 24 2020