Upload Images Directly to S3 from Front End

in this video i want to show you how you can upload images directly to an s3 bucket and then retrieve them from the s3 bucket later rather than having to go through your server to actually upload and download the images and a benefit of doing it this way is that the load that you put on your server is significantly decreased because it never has to process image data and forward it to and from an s3 bucket so one way of doing this is to upload the image first to your server and then have the server push that image directly to the s3 bucket then later on you make a request for your image to the server and the server has to get the image from the s3 bucket and then deliver it back to the client a different method is to instead post the image directly to the s3 bucket and then when you need the image you get it directly from the s3 bucket and the server never has to handle the actual image data which significantly decreases the amount of data transfer going to and from your server running your internal logic this also makes it a little bit easier to start using cloudfront a cdn that you can connect to an s3 bucket so you can distribute your images globally and then users request the image from the location closest to them rather than from the location of the s3 bucket there is a tiny bit of server code needed to make this work and i'm going to be using node to write that server side code however this pretty much applies the same to any backend language this is basically a three-step process where we first have to make a request to our own server to get a secure url for the s3 bucket so our server will have credentials that allow it to access the s3 bucket and it will create a secure temporary url that it sends back to the client the second step is that the client actually has to post the image data directly to the s3 bucket using this secure and temporary url and then once that is successful once the image has successfully been uploaded to s3 the client should then tell the server that the request was successful and send any other bits of data that the user needs to provide so if our file upload came with a description and any other details we could then send that to our own server to store in a database separately so right now i have a really basic html file that has a form that needs to upload an image and it has an upload button and then in my javascript i'm going to handle this uh by listening for a submit event on that form and then i need to get that file data that was added to the form in the html file to my s3 bucket and the way this works is that first we have to get a secure url from our server to then post the image directly to the s3 bucket and then once that's successful once it's actually been saved to s3 and i've got a confirmation i could then make another request to my server so post request to my server to store any extra data so the first step is to get that secure url and that's going to happen on the server and right now i just have a basic express server set up what i'm going to do is create a new endpoint that will allow my client to get a secure url [Music] and i'm just going to call this s3 url for now and realistically you'd probably want to authenticate and authorize that the user is able to upload images but just a basic implementation i'm just going to use this to get a temporary url the next thing i'm going to do is actually create a new file on my server this is part of my server-side code i'm just going to call this s3.js and this is where i'm going to put all of the code that connects to aws and the s3 bucket and handles the generation of the secure url [Music] so so far all i've done is imported the aws sdk for node into this file and i've set up some variables to keep track of the bucket name the region that the bucket exists in the access key id for the bucket and the secret access key for the bucket i'll get to those in a second and then i'm creating a new s3 bucket object using these details so really before i move on i want to be able to set all of these things up so we actually have an s3 bucket to push the images to in the future so the first thing i'm going to do is actually create a brand new s3 bucket and to do this i just go to aws.amazon.com and sign into the console if you don't already have an account you should create an account and sign up then i'm going to click on the s3 service in the console and it'll take me to all of my s3 buckets and for this i'm going to create a new bucket and this is just object storage where i can store a bunch of images in the cloud and i'm just going to leave this as us west 2 that's going to be the region i create my bucket in and i'm going to call this bucket something unique direct upload s3 bucket thing i don't know and then i'm just going to add this to my list of variables so i don't forget about it so i'm going to end up with a bucket with this name in this region and then while we're in aws we can configure this bucket so we are going to allow public access to this bucket we're not going to block any of the public access things because we need the web client to be able to add images to this bucket and then we're going to allow everyone to be able to get images directly from the bucket using the public s3 url so i'm going to acknowledge that i'm allowing this bucket to be accessed publicly it's not going to be private and i'm going to leave everything else as the default settings then i can just create this new bucket once this bucket has been created i'm going to go back in because i have to configure a few more settings so in the permissions tab we have to go down to the bucket policy and actually edit this policy to allow anyone anyone on the public internet to be able to get an image from this bucket we're not going to allow them to post or modify or delete we're going to require our server to be authorized to be able to do that but we are going to allow anyone to be able to get the image that will allow the web client to actually uh just put the image url in an image tag and be able to display it on a web page so to do this we have to edit the bucket policy and we can actually go to the policy generator to generate this for us so from the drop down we're creating an s3 bucket policy uh what we're trying to do is allow so we're going to allow uh anything from anywhere to uh get there we go get object so this will be allowed to get a single or many images from the bucket and we're going to add the arn of the specific bucket that we want to allow this for the arn format is arn colon aws colon s3 colon colon colon and then the name of the bucket so allow anything to be able to get images from this bucket basically and then if we generate the policy here it's just a little bit of json we have to copy this we can close this window and back in my s3 page i can just paste this in this will now allow public read access of the objects so we can save the changes oh this actually gave me an error oh because we're not allowing read on the bucket itself we're allowing read on the objects within the bucket okay so save changes there we go just that slash star means uh you can access the images in the bucket or the files in the bucket but not the actual bucket itself so we have the bucket policy updated that'll allow us to get images um there's one more thing and that is cores we can already get the images and download them from the s3 bucket but in order for a web page to be able to display these images in an image tag for example it needs to have permission to make a request to a url that's different to the one it was served from and this is going to be an s3 url so we have to edit the course policy here and in a production app we would want to edit the course policy to only allow requests from our domain name uh but for development it's easy to just open up this course policy to allow get requests from literally any website so anything can make a get request to rs3 bucket to get the images i just have the course policy uh copied so i'm just going to paste that in here basically we're going to allow get and head so that anything can make get requests and get the images we're also going to allow put because that's the method that our http client is going to use to actually put the image into the s3 bucket and no one will be able to do this unless they have a secure url but we do have to allow that in the course policy so if i save these changes this bucket is now set up so that we can get images and post images to the bucket so getting the images is going to be really basic really easy anyone or anything should be able to do it but adding a new image to the bucket is something that only our application should be able to do that maybe only authorized users are able to do so in order to allow put access on the bucket we're going to make a new iam user so if we click on the iam feature we're gonna have to create two different things we're gonna have to create a policy and an actual user so a policy is a thing in aws that specifies what type of things a user can do so we're going to create a policy that says this user is allowed to put an image into this specific s3 bucket and then we'll create a user with that policy and our server will use the user credentials there to actually be able to connect to our s3 bucket securely so the first thing we need to do like i said is create a brand new policy uh the service we are creating the policy for is s3 we're going to allow put access so list read write going to allow put object for that specific bucket so we put the bucket name in here and i'm gonna say any for object and this is just saying uh we're gonna allow something whatever has this policy to add a new object to this bucket and that's it so we can go to tags review i don't really i guess i should add a name i'm just going to use the same name i use for the bucket that'll make it easier to find later and i'm going to create this policy so now i need to create a new user and i'm just going to give this the same name as the bucket again i'm just using the same name so i can easily find all this stuff later and what we're going to do is create a programmatic access user so that we get an access key and a secret key and we're going to give that to our server so our server can connect to the s3 bucket securely so i'll go to permissions and here we're going to attach the existing policy that we just made and go to tags i don't care review all right and then we create the user so i now have a brand new user and this is a user just for this application to connect to that specific s3 bucket and i get a access key and a secret access key here so what i'm going to do is i'm going to copy both of these and i'm going to go back to my application and i'm going to create a m file and this is where i'm going to store all of my secret details [Music] so i'm going to close that file now and when i need to access those details i'm just going to do it through the environment variables and then i need to npm install the dot m library to make this work so in my development environment i'm gonna have a dot m file and then on my production server i'm gonna actually have these environment variables set up on that server just need to import dot m from m and configure it so i now have my s3 bucket set up in this region with this name this access key id and this secret access key so now my server should be able to create that secure url that my client can then use to post the image to the s3 bucket [Applause] [Music] [Applause] so here's the generate upload url function that i'm going to use i start by creating an image name and this should be some sort of unique name that won't collide with any other images on the server and this obviously isn't good enough but i'm gonna come back to this in a second uh then we specify the param so this is the bucket name the image name and the expiration of the url so the url will only be valid for a certain amount of time we send that back to the client and the client has this amount of time in seconds to actually post the image to the s3 bucket before the url expires so i'm just going to specify 60 seconds that's how long the url is valid for and if it isn't used within that time then the client's gonna have to request a new url then we call s3.getsigned url uh the promise version and we're just saying we want url so that something can put an object to that bucket and then once we have that url i'm going to return it here and then we'll be able to use that from the express server uh so for this random image name really any random value would be fine but if we want if we want to make it a little bit difficult to guess what the urls are going to be there is one technique i've seen that i'm just going to use here [Music] so we're going to use the crypto library from node and we're going to generate 16 random bytes convert that to a hexadecimal string so it's going to be a 32 hexadecimal character string that we're going to use as the image name and this will uh give us a unique string that's really difficult to guess so there's some level of security there so people won't just be able to uh get every image from our s3 bucket and now i just need to use this function in my server so if i import s3 when i request that new url i'm gonna call [Music] and then just send it back to the client so now from my client side code when the user submits that form the first thing i need to do is make a get request to my server to get that secure url [Music] so i'm going to get the url from the server and right now i'm just going to console log it out to make sure that everything's working so if i open up the console here and something is already failing didn't import that correctly let's just fix that okay the service running okay so if i refresh this page we should see the url printed out here when i submit the form actually i don't even need an image there i'm just going to click upload it cannot find localhost 8080 s3 url because i don't know how to use express there we go okay let's try again click upload and we get nothing back from the server because i'm not awaiting that okay now uh the server failed because random bytes is not defined that is right i was supposed to create a random bytes function that equals promisify crypto dot random bytes okay hopefully that's the last mistake upload damn un known signing version aha it's supposed to be version four okay that's the last mistake though right yes okay cool so i'm just gonna refresh and do that again if i click the upload it's gonna make a get request to my server my server is gonna make a secure connection to the s3 bucket to get a secure url and here is that secure url so now my client needs to take that url and make a put request to the s3 bucket using that url to store the image in that s3 bucket and currently there is nothing in the s3 bucket so if this goes well we should be able to see an image here so back in my client side code here once i have the url i now need to make a put [Music] request [Music] so we're going to take the url and we're going to make a put request to that url using content type multi-part form data and we'll just specify that image file that's coming straight from the form uh in that put request and we'll wait for that to happen and then once that's successful we can get the image url just by splitting the put url on the question mark and getting the first part of it and that should give us the actual image from the s3 bucket so let's test this out and see if it works so if i choose a file let's see it's a good one and click upload we get the url it sends it to s3 and this is now the link to the s3 bucket uh if i refresh the s3 bucket i should be able to see that that image is in there fantastic and there's that unique name it has uh and if i click on this link it should uh well i want it to show the image um but really because it doesn't have an extension the browser will just download it directly which kind of sucks so what i'm gonna do now is i'm just gonna every time an image gets uploaded i'm just going to display in an image tag just so we can see that it's actually working [Music] so now if i go to post a new image i should see that image just appear here so i'm going to choose that image again here's a different one upload goes it makes it to the s3 bucket and there it is on my web page and if i check the s3 bucket again there should now be two images in my bucket there we go so the final part of this if this were a real application that i was working on would be at this point once the image has successfully been posted to the s3 bucket i need to tell my server that was successful and i probably have more data with my image like a description or any other details so i'd then make another fetch request to my server to give any other details after the image was successfully uploaded so that is how to save an image to an s3 bucket directly from the client using a secure url generated by the server in a future video i'm going to build on top of this to actually set up a cdn with cloudfront so that your images get distributed globally and your clients request your image from the location closest to them

Channel: Sam Meech-Ward

Views: 16,730

Keywords: aws, s3, node.js, amazon s3, s3 bucket, s3 tutorial, web development, image upload javascript, node js, javascript file upload, javascript aws s3, react image upload

Id: yGYeYJpRWPM

Length: 18min 27sec (1107 seconds)

Published: Thu Apr 08 2021