Storing Images in S3 from Node Server

and that's a pretty complete app now i can create a new post store the file in an s3 bucket store the details in a database i can then get those details back and access the image using a signed url and i can delete the image and the post from the database when anyone wants to actually delete their posts [Music] in this video i'm going to show you how to use an s3 bucket with a node.js server so that you can store files for your web applications and i have code examples for an ejs app a react app and a next.js app and i'll link to these in the description but in this video i'm going to be focusing just on express and react but really i'm only focusing on how the server interacts with the s3 bucket and it could be any client-side application interacting with the server for this video i have this demo application called instasam which is kind of a knockoff of instagram which will allow us to upload images to an s3 bucket modify those images delete those images and obviously download these images so that we can see them in the browser and basically a user can go and create a new post select a new image file add some sort of caption and submit this to the server and this will store the image in s3 create a post in the database and then allow us to view these posts in the client side application this is the finished version of the application and in this video we'll go through all the steps to get the server side working with s3 and i'm only working with image files in this video but the code we write will work with all files so you could take this code and use it to upload videos to s3 and from a big picture this is what it's going to look like to create a new post the client will upload a photo and a description to an express server the server can then validate any of the details make modifications to the photo like resizing it and then the server can send the photo to an s3 bucket where it will store it then we can store information about the photo and the description into a database when the client requests a post the server will get the data out of the database generate a secure unique url for the photo and return that data back down to the client the client can then present that data and use that url that was generated on the server to get the image that's stored in the s3 bucket then if the client wants to delete a post it just sends a request to the server that can delete the photo from the s3 bucket and the entry from the database if we take a look at the code that's currently set up for this it's a really really basic express server i'm using prisma to interact with the database and i have three endpoints a get request to get all posts a post request to create a new post and a delete request to delete a post and as you can see none of these are really implemented yet they're just placeholders for the code that we're going to write and if we take a look at the client side code here this is all the react app this has all been fully implemented we're not really going to add any code here but we can take a look at what the front end code might look like so i'm going to go and run the back end server and the client-side react tab here so they're both set up and i'm going to go back over to the web browser and i'm gonna try and create a new post and remember the code to create a new post has not been implemented yet so it's just gonna be this code running and i'm gonna console.log rec.body here just so we can see what kind of data is being sent to the server when we make a post request to slash api slash post so i'm going to select a file uh and add a caption here spelt incorrectly and i'll open up devtools so we can see what's going on with the network request so i'll submit this form to the server and we should see that it makes a post request to slash api slash post a post request right there and if we take a look at the payload we can see that it is sending an image as binary data and the caption as a piece of text and if we take a look at the headers there is one thing that is important and it's the content type of the form that is being sent so here we can see it's multi-part form data and this is really important because to send an image file to a server the form has to send it as multi-part form data so if we take a look at that as what it would look like in a plain html form we would set the ink type to be multi-part form data and in a react app we have to use a form data object to send that to the server so it's important that that gets sent as multi-part form data which it does and then on the server side we're logging rec.body out to see what kind of information we get from that post request and in the terminal here we can see that rec.body is actually undefined and that's because an express server doesn't know how to deal with multi-part form data by default anyway so we need to install a middleware function that will handle the multi-part form data for us and the middleware that we are going to use is called malta so we can install this just by running mpmi malta so i'll do this on the server side code and now i'm going to head back over to the malted documentation because it has a few code examples that we can use to get started pretty quickly and this first code example shows how to upload a single image and store it on the server's file system and we can do this and then upload the image to s3 but what i want to do is actually use the memory storage so we just keep the image in memory we can modify it in memory validate it in memory things like that and then we just push it straight to s3 without ever saving it to the file system on the node server so i'm going to search for the memory storage here and here it is so i'm going to copy these two lines and i'm probably going to need some of the code from this code example 2 because i want to use this middleware and then i'll just require it manually so i'll explain this as i go the first thing and i'm using es modules here is i need to import malta from malta then what i want to do is create a memory storage object and create an upload function with malta that is just going to make sure it always stores the image in memory it's just never going to store it to disk once i have that i can create this middleware function which is just going to look like this upload.single because i'm just uploading a single image if i wanted to upload multiple images i could just check the documentation and change this so that i could upload multiple at once right now i'm just going to keep it as single and the string that goes in here has to be the name of the image that i'm uploading so if i check the network settings again and check the payload of what i'm uploading i called my image the word image so i'm just going to use image in there and the way you would see this is if you're using an html form then it's the name of the input right here so the name is image in my react app it's whatever name i put in here so whatever the string is here has to match whatever you put in that middleware function so upload single image and then i have to add this as a middleware function to the route that i'm uploading to so i'm uploading to slash api posts so this is going to have to just go in the middle right here and once i have this and i make a successful multi-part form data request here i should get any details in my rec.body that aren't to do with the image so in this case i have a caption which is a piece of text that will appear in my reg.body but all information about my image will appear in a rec.file object so i'm going to log these two out right now so that we can see what we get in the body and we can see what we get in the file so let's see what we get when we run this now so i'm going to run the server and then i am going to just create a new post again and if we head back to the logs we can see here's the caption that i put in that's the rec.body and then here is all the information about the image that i uploaded and here is the buffer and this is what we need to access the actual image data modify any image data and send that to s3 so rec.file.buffer is the actual image and everything else is just data about the image so this is the thing we need to send to s3 and before we can actually send it to s3 we need to set up an s3 bucket so right now i'm going to go to aws.com and log in to my aws account if you don't have an aws account go sign up for one now if you do then log in and just follow along with me as we set up the s3 bucket once you're logged in navigate over to the s3 dashboard and all we need to do here is create a new bucket so we'll click the create bucket button here and you can give this any name you want just as long as it's a unique name so i guess i'll go insta sam demo thing that's probably unique i'm going to throw this in canada because that's where i am and that's it everything else can be the default settings but make sure that you probably just actually copy the bucket name right now because you're gonna need to remember the name you're gonna need to remember the location everything else can just be the default settings everything's private by default so this is secure and we can just create the bucket like that so i have this new bucket i'm going to open this up and just leave it open and if i look at the properties again i can see the name and i can see the location it's in and right now before i do anything else i'm going to put these details into my server side.emv file so i have a dot emv file already set up with my database url in it and i'm gonna put in my bucket name which is instasam demo thing and my bucket region which is c a central one and i'm just going to change these double quotes to single quotes and then the other information about the bucket will also go in here once we make it so the s3 bucket is basically just a big hard drive in the cloud like when you buy a usb hard drive to plug into your laptop because you need more storage for your laptop and you get like a terabyte for like 300 bucks it's fairly cheap that's pretty much what s3 is it's just a big hard drive it's really really cheap and it's just for storage we can't run any code on it we just store files on s3 so we're going to have our web app that's going to be deployed to the web every time it needs to store a file it stores it in s3 and it will just scale up to as many files as we need and it's a really cheap way of storing files it's also necessary to use something like s3 if we want to horizontally scale our web apps and i have more videos on aws and infrastructure and cloud computing so i'll leave links to the relevant ones in the description if you want to check out more on aws and cloud computing but for now we're just gonna do the bare minimum we have created this bucket by default buckets are completely private nothing can access this bucket nothing can put files into the bucket nothing can get files out of the bucket except for me because i'm logged into my account i just created this bucket i can go and i can put objects into this bucket put files in here i can get them back but nothing else can so what we need is for the express server to be able to log into the aws account and actually put files in and get them out for us and to do this we head over to the iam console so i'm going to open up iam in a new tab here and there's two things we're going to need to make right now one is a user we're going to create a user that will represent the web app that will allow the web app to access the aws account and access the s3 bucket the other thing we need to make is a policy and a policy is the thing that defines the rules for what can be accessed by a user so i want my express application to only be able to access that s3 bucket and only access it in very specific ways so the first thing i'm going to do is set up a policy to define exactly how it can access the s3 bucket so i'll click on policies here and then create a new policy up in the top right there and the service we're creating the policy for is an s3 bucket and then we get to define the actions that the web app will be able to perform on the bucket and we really only need a few actions here the first one is a read action we want the web app to be allowed to get any file from within the s3 bucket so get object we need to check then for writing we want to check two of these the first one is going to be put object this allows the web app to actually put a new file into the s3 bucket and the other one is delete object because we want it to be able to delete a file from there and these are the only three things that we're going to allow our web app to actually do with the s3 bucket then in this next section in the resources section we're going to specify which bucket it's allowed to perform these actions on so here i'm going to click add arn and we're going to add the arn for the bucket that we just made and all we need to do here is put in the bucket name so that's the name of the bucket that we just made and then i want this web app to be able to put get and delete any object within that bucket so i'm just going to check any here and then click add and we can see the arn got added in here and that's all we need so s3 bucket we're going to allow get object delete object put object only on that bucket any object within that bucket that we just made so then we can click this next button here to go to tags i don't actually care about adding any tags so i'm going to click next review and then here we have to give it a name so this is specifically a policy to access the s3 bucket for the web app so since my web app is called instasam i'm going to call this instasam s3 bucket access maybe something descriptive for what is going on here create the policy and now that's done it's just added into my list of policies so the next thing i need to do is create a new user that is going to represent the web app to access the s3 bucket so i'll navigate over to users here and then create a new user by clicking add user and a user can be a real user like someone that is a person that uses a username and password to log into my aws account or a user can be a application that accesses things within my aws account using programmatic access keys so for this express application it's definitely not a real user it's an application so i'm going to check the access keys programmatic access checkbox here i'm going to give this user name the name maybe instasam web application that seems good enough so this is going to be an application accessing things on my aws account so right now when i create this user it won't be able to do anything until i attach an existing policy to say what it can do so i could attach something like administrator access and my web app would be able to do absolutely anything in my aws account that'd be a really bad idea instead i'm going to search for that instasam s3 bucket access policy so this user will only be able to do those things that we defined in that policy earlier click next add tags i don't care about tags i'm going to review this and then create the user so this will create that user it will be able to access the bucket and right now on this page we need to access two things first is this access key so i'm going to copy this and head back into my environment file and create a access key variable that contains the access key and then the secret access key which this long string here and this is a secure key it's kind of like a password so you shouldn't ever show anyone this key or share this key with anyone i'm going to delete this uh imuser as soon as this video is done so i'm not super worried about this and i'm going to call this secret access key in my environment file so now i have all the information i need for my web application to be able to access images within that s3 bucket and to do this from the node application we're going to need to install the aws sdk and currently this is separated into the different libraries that can be used so we're going to be looking for the s3 client and you can see a bunch of documentation and i'll link to this in the video description um but basically we're going to be using this library to interact with the s3 bucket so i'm going to copy this first line of code here head back over to my server and i'm going to be importing the s3 client i don't care about the second part here but the s3 client is what we're going to use to interact with s3 and since i'm using environment variables now i'm going to need to import the dot m library from dot m and then called dot m dot config so this will bring in all the environment variables and i'm actually going to just copy these for over so that i can easily create variables for them so let's go const bucket name equals process m dot the name um see how quickly i can do this we'll transform that to lowercase and then everything after the underscores can be transformed to uppercase and then i need to delete the underscores but not from the environment variables over here but there we go okay cool so this just gives me access to all those environment variables as javascript variables and now i need to configure my s3 client using these variables so i create a new s3 object see how much i'm going to get from autocomplete here awesome but it actually got that a little bit wrong because i need a credentials object right here that wraps the secret access key and the access key id so we can set up a new s3 object using the credentials from the environment variables file and the region of the bucket so down here where we're trying to actually store the file we want to send this data to the s3 bucket and to do that we need to create what's called a put object command so back where we imported the s3 client we also need to import put object command there we go and down here before we can actually send the file we need to create actually create this variable const command equals new put object command and here is where we specify all the information about the file so i'm going to create a new variable for this called params params equals and we have to specify the bucket name let's see what we get from autocomplete ah the key which is the file name the body which is the file buffer and it's really good to specify the content type which will tell s3 what kind of image it is so this is gonna currently try and upload to the bucket that we made it's gonna use the original name of the file which is whatever the file was named on the user's computer before they uploaded it we're going to send as the body that file buffer data which is the entire image and then we're going to set the content type there so we have this command now we just have to tell s3 that's the object that we made right here with all of those details we have to tell s3 to send that command to the s3 bucket so we're gonna write it like that and then this is asynchronous so we'll await that and that's it so if that is successful if we get to the end of this we get a successful response we should have a new image in the s3 bucket so i'm going to go back to the s3 bucket here i think it was on this page and i'm going to refresh this page just so we can see we have no objects in here if we go back to the react app and create a new post again i'm going to choose the same file as before so this is called patrick i think it's patrick.png so this is what we should see in the s3 bucket once i upload this it doesn't matter what i type in here just yex we're not saving anything to the database but i'll submit that it looks like that was successful okay all the logs on my server don't suggest there were any errors so now if i go back to s3 here and click refresh i can see there is a patrick.png image and if i click on this and download the image and then open this up we should see there is the actual image so this image is going from the react client to an express server to an s3 bucket and then i can download that because i'm logged into my account here so that is creating a new image in s3 and if i keep doing this i can do this again i can create a new post add a new image in here let's go sam submit we will see that i should have now two images in here patrick and sam then if i go in again and choose file patrick open submit and re-upload a file with the exact same name i mean it happens to be the same image in this case but really the only thing that matters is that it has the same name if i look on the server i can see that there are only two images patrick and sam and patrick was modified more recently than sam and that's because every time we post an image to an s3 bucket if the file that we're uploading contains the same name as an existing file it's just going to completely overwrite the old file with the new file so if you had users for a web app uploading images and some of them had images with the same name you would end up completely overwriting one of the user's images so in cases like this it's really important that your key here which is your image name is something unique that won't cause a naming collision on s3 because chances are every time someone uploads an image you want to store that as a brand new image on s3 so right here we need to create a unique file name and we can do this in a couple of different ways but it can be handy to have a unguessable image name to keep things a little bit more secure so the way i'm going to do this is i'm going to import the crypto library from crypto and i'm going to use this to create a quick function here that's just called random image name and this will just be a function that uses crypto to create random bytes and converts it to a string and i'm actually going to let this be configurable so we got bytes that will default to 32 bytes and i'll convert that to a hex and real quick just to show you what this is doing i'm going to open up a node ripple and i'll import crypto and run this line of code and actually change bytes to 32 and we can see it just generates this random hexadecimal string that should be fairly unguessable so i'm going to close this and go back and i'm going to use this to create random image name so every single time we go to upload a new file to s3 the file name will be completely random i'm going to call this function to create a random file name so if we go back and upload a new image now i'm going to select an image that has existed patrick again and submit that if i head back over to s3 and refresh i can see that that image was added with that completely unique name so i could keep uploading this image and it would never overwrite the existing images however if you did ever want to update an image all you would have to do is run that exact same command that exact same put command from the express server right here and just make sure that the key here the image name is the same as the image that you want to overwrite now the image is being successfully uploaded to s3 let's take a look at how we might actually modify the image before it gets sent there so what i'm going to do is i'm going to install a library called sharp that allows us to easily do a lot of image resizing and things like that and here is the documentation on how you can use sharp what i want to do is resize the image so that it basically looks like a tick tock video or an instagram reel so i'll import sharp then down here i'm gonna call sharp and i can just pass in the rec.file.buffer i'm gonna resize this to 1920 by 1080 so that it looks like a iphone in portrait mode i will convert this to buffer again so that we can get that image data back and send it to s3 and i'm going to make sure that this doesn't stretch by setting its fit to contain and i actually think this whole thing needs to go into an object where i set the height and the width like that and then i also need to await this because it's an asynchronous call and get the buffer from this function which i'm going to then send into my body for my s3 request so i get my file from the post request pass that into sharp do some things with it get a new buffer of the modified image data and then send that to s3 so i should get a resized image at the end so i'm going to make sure my server is running and then i'm going to try and upload an image again and what we should see is that the image has been resized by the time it makes it to s3 so i'll submit that go back to s3 i'm looking for the most recent upload that's i think this bottom one so i'm going to download this and open it up and yep there we go so i now have my resized version of the image so right now by uploading the image to our server and then pushing it to s3 it makes it really easy to do things like modify the image before it ends up on the s3 bucket and the only thing left to do now is to store this information in the database because right now i'm just saving an image to s3 which is great but i really need to store all of this post data like the actual caption that the user sends and any other information into the database so what i'm going to do is after only after it's successfully been posted to s3 i'm going to that's nice create a new entry uh using prisma that contains my caption uh which is rec.body.caption and my image name which is this random image name that i'm creating here so i'm going to break this out into a variable and then store the image name in my database and this is all i really need i need to be able to save the image name because that's how we'll retrieve the image later from s3 and then any other information that i want to store about the image or about the post and in this case it's just a caption so this should be good i'll create that in prisma and then i'm actually going to send this post back down to the client so if this is all successful we should have a new post created in the database uh so let's try doing this again i'm going to choose a file uh this patrick one call it patrick and submit this and if this is successful this time hopefully it's successful there we go i'll see that the response was actually the object that got created in the database which is awesome and we can see that it's actually displaying this now on the web page we're not displaying the image because we don't have access to the image yet we haven't gone through those steps yet but this has created the post it does appear in the database i'm actually going to load this in the database right now just run prisma studio so that we can see here are the posts and here's the data that got saved in the database so there's the image name there's the caption i just have some other information about the post that i wanted to save just to make it more realistic but there is the information being saved and that's great this all works now we just need to figure out how to actually access the image so that when we load the posts on the page like this we actually see the image so remember everything in this s3 bucket all of these images are completely private i can see them now and i can access them now because i have access logged into my aws account and my express server also has access to this bucket but nothing else does so no one can actually access these images no one using the web app can see the images only the server and myself have access so what we're going to do is generate urls from the server that will allow users to see the images for a temporary amount of time so these urls can expire and we can generate different urls for different users and only generate a url for an image if a user did have access to that so if i implemented some sort of following system you could only see images potentially of users that you follow on instasam and to do this we're going to be using a library called s3 request pre-signer which again is another aws library has a whole bunch of documentation here first thing that we need to do is install the library and this top example seems to be exactly what we need so i'm actually going to copy all of this in and what i need to do is first bring in the get signed url function from the library i'm going to import the get object command from the s3 client because we're going to be using that to get the actual object and then i'm going to take the rest of this stuff and i'm going to put it in my get request so when a user makes a get request to get all the posts this is just going to get a list of all the posts that exist in the database we're going to generate urls for those posts so right now you can see the data for the post on the right and it contains image name and caption what i'm going to do is add an extra field in there called image url and that is going to be the signed url that we're going to make right now so when we get a post first we're going to get all of the posts from the database because we need those image names and then i don't need this client part here i just need these parts so we've got a get object command that we're going to use to create the url and for this we need get object params so this is going to be an object that contains the bucket name and the image name of the image that we're trying to retrieve so since i have an array of posts here i'm gonna have to loop over for posts of posts i'm gonna have to loop over all of the posts and generate a url for each post that i'm getting so the key here is gonna be the post dot image name so within this bucket for this exact image we want to generate a signed url and then i'm just going to attach this to the post so post.imageurl equals url and if i go back into my browser and i'm just going to make a request directly to my server here so port 8080 slash api posts but i need to make sure that my server is actually running so let's run that then head back and refresh this client is not defined that's right cause i copy and pasted that code client is gonna be s3 which is the s3 object that we configured up here which has all of those details in it so now if i refresh this comes out i guess i don't have a json parser in my browser right now um so maybe i'm actually going to do this as a curl request local localhost 8080 slash api slash post pipe that into okay that also doesn't look great but we can see that here is my array of objects currently i only have one post in the database but it has that url right here https instasam demo thing dot s3.ca central so here is the actual object in the s3 bucket here's how i could access it but it can't be accessed directly because we wouldn't have permission to access anything in the bucket but by adding all this extra stuff in the end we've granted temporary access to this image so if i take that url and i just load that into the browser here we should be able to see this image and this is because we're using the signed url if we look back here we can see there's an expires in section this is 3600 this is in seconds so this is going to expire in an hour if we wanted it to expire much sooner maybe we generate one for just in 60 seconds that's completely up to you the idea here is that we're allowing access temporarily to this image which is fine because if the user still needs to access the image they just make a request to the site again so if i refresh this page now we can see that the user is able to access this image and because this is all happening on our server side code the way that we generate the url we could very easily check to make sure the user is logged in make sure they have permission before we give them any access to any url to get any images that they should or shouldn't see and the reason that this is working immediately is because in my client-side code i have just already got it set up where i'm expecting a parameter called image url and i'm just dumping that straight into the image source tag so all you need to do is take that url put it in an image tag and then it will just work so that's creating an image and getting an image but now if the user clicks this delete button down here my client is going to make a delete request to my server for that specific post and what i want to do in this case is delete the image from s3 and once that's successful delete the entire post from my database so right now i'm doing nothing on the server side code i think my delete request is just a placeholder so on this delete request we need to be able to delete the image from s3 which means we need the image name of the post that we need to delete so the very first thing that we need to do here is actually get the post from the database so post dot find unique where id there we go all right that's perfect so first i'm going to find the post from the database and then i'd probably do some sort of check to make sure that there is a post uh if there isn't maybe we'll just res dot oh look at this let's just do the autocomplete cool return um if we have the post then i know the post dot image name is the key of the image that i need to delete from s3 so now so i need to create an object called params that contains the bucket name and the key which is right there in order to complete it for me uh and then i need to send that to s3 in a not a get object command but a delete object command will that import automatically for me yes it did okay so we need to import the delete object command and we're creating this command this time and then just as before i'm going to await s3 dot send and just send that delete command so same kind of details as before same code as before but the command is a delete command instead of a put command so get the object from the database delete it from s3 once we're done with that i'm just going to delete it from my database as well and then i guess i could just send back the post to the client if everything is successful it's deleted cool but those are the steps find it delete it from s3 delete it from the database so if i go back into the browser now if we refresh my s3 bucket i should see there's one two three four five different images in s3 and i have currently one image on my client here so i'm gonna hit that delete button and hopefully it should go through all of those steps find it delete it from s3 then delete it from the database and if that's successful if i refresh my s3 bucket i should only have four images which i do right now if i check prisma again i should see that this is completely gone from the database and everything has just been deleted so if i delete a single post everything just works as it should and that's a pretty complete app now i can create a new post store the file in an s3 bucket store the details in a database i can then get those details back and access the image using a signed url and i can delete the image and the post from the database when anyone wants to actually delete their posts but every time i upload an image it's getting stored in my s3 bucket in central canada which means that anytime someone accesses the image they're always going to have to access it from canada even if there's somewhere else in the world like europe or australia so static files are usually better distributed over a cdn which is kind of like s3 but copied globally so that you send the file from the location closest to the user so if someone makes a request from somewhere in europe they're getting that image from a really close location to them in europe or australia or china or wherever and since we're already using s3 and aws we can integrate the s3 bucket into a cloudfront distribution on aws and this will just automatically set up a cdn copy over our files globally for a really cheap price still but allow much lower latency when someone requests a file from our site so in my next video i'm going to pick up where i left off here and show you how to set up a cloudfront distribution so that you can distribute your images this way but over a cdn instead of an s3 bucket so make sure you're subscribed to this channel and hit that bell icon so you don't miss when i make any new videos like that and if you like this video leave me a comment let me know what you thought and let me know if there's any other topics that you'd like me to cover in other videos [Music] you

Channel: Sam Meech-Ward

Views: 40,632

Keywords: uploading images to s3 bucket using nodejs, uploading files to s3 bucket using nodejs, upload files to s3, aws s3, s3 bucket, s3 bucket access key and secret key, s3 tutorial aws, node js express project, upload files to s3 bucket nodejs, upload file to s3 nodejs, upload images to s3 nodejs, how to upload image in s3 bucket using node js, uploading files to aws s3 using node js, upload file to s3 bucket javascript, upload files to s3 bucket, Upload Images to S3 from Node

Id: eQAIojcArRY

Length: 39min 58sec (2398 seconds)

Published: Mon Jul 25 2022