Upload Images to S3 from Node Back End

in this video i'm going to show you how to connect your express server to an s3 bucket so that it can store all the images in the bucket and then get them back later when the users need to see them and the way we'll accomplish this is by uploading the image to the express server first and then pushing it to the s3 bucket then when the user needs to request a file it will first go to the express server and then you'll get it from the s3 bucket and send it back down to the client this gives the node app complete control over everything so that if we only want to show images to users that are authorized we can control all of that with javascript i have a separate video explaining how to set up s3 buckets just to serve static content and an article explaining how to just upload images to an express app without using an s3 bucket and i'll put links to those things in the description i have a really basic react app here that just displays a form to a user so they can select an image give it a description and submit it to the server and if we take a look at this this is the basic form and if we click submit it tries to make a post request to the server at slash images to post the photo data and right now the server is doing pretty much nothing it's just sitting here allowing post requests images and doing nothing but sending a response so the first thing we want to do is actually have this endpoint right here allow images to be uploaded and store in the file system on the server and to do this we have a really nice library that we can use called malta and this library lets us receive images on the express server and store them in the file system by writing like three lines of code i think so the first thing i have to do is actually install malta and then pretty sure i can just copy and paste some of this example code got a love example code so this will require the malta library and then i'm going to specify a destination for all the files that are uploaded to my server i'm just going to store that in an uploads folder and then the final piece of the puzzle here is to create a middleware function that will accept a file and in this case i'm going to call mine image because on the client i am uploading the file and i'm calling it image in the form data and now when this post request is made uh malta will check for an image binary data in the data being sent to the server and store it in the uploads folder so if i actually go back to the application on the front end and try uploading an image let's take one of these and submit it if i look at my post request it posted to slash images it sent the image as binary data little description not really going to use that for anything just wanted to show that we can send any data including an image and if we take a look at the server file system i'll see that there is an uploads directory that's been created and the image has actually been uploaded to the server and is just being stored there so getting the image to the server is a really easy step and malta also gives us a few pieces of data here the data about the file is stored in rec.file and any other information that was sent up to the server is going to be stored in rec.body so in this case i sent a description so i could just pull that out of reg.body if i wanted to i'm not actually going to do anything with it but just nice to know i have that um and i'm just going to console.log the file object so we can see what kind of information we get from that so if i submit that photo again it should upload another photo into the uploads directory and if i check the console here we should see this is all the information we get about the image so there's the path that it exists on the server upload slash that random name that malta gives it uh the actual name of the file few other bits of information i don't really care about right now but this path and file name are going to be useful because the path is where the file actually exists and we're going to need to send it from there to the s3 bucket and the file name is handy because we can just keep using that same file name in the s3 bucket and then if we had a database set up we could store that file name in the database so that we could easily get back the image later on so at this point files are being uploaded to the server and being stored in the service file system and now we just need to get them to the s3 bucket before we can actually send an image to an s3 bucket we need to have an s3 bucket set up and we do that through our aws account so if you haven't already you'll need to sign up for an aws account i already have mine set up so i'll just log in and then once you're signed in you can navigate to the s3 bucket console and here it's really easy to just create a new bucket you go to create bucket and give it a name this has to be a unique name so let's go with uh insta amazing appstagram there we go that seems unique and then i'm going to store it in north virginia you can choose any region you want but if you're not sure which region just go with north virginia and that's it we can leave everything else as the default settings everything will be kept private and secure and we'll just create that new bucket and now if we go and find the bucket we can click on the bucket and we can see that it currently has no objects in it and we can modify some of the settings from here if we wanted to we don't we want to leave it alone but there are two pieces of information we want to store that we'll need in our application the first is the bucket name so i'm just going to copy this from here and go back to my server file and i'm going to store all of these things in a m file because we won't really want to commit a lot of this data with the application we want it separate so i'm going to call this bucket maybe aws bucket name uh aws buckets region this is us east one for north virginia um so that's it for now we'll add more data to this in a second so we have the bucket we want to be able to post images to the bucket but right now this bucket is completely private i can access the bucket right now because i'm logged in as a user in my aws account and my account allows me to access all of the services within aws that are owned by my account but no web app or no other user can access this s3 bucket so we need a way of providing permission to our express server to actually access this bucket to upload and get images from the bucket and to do this in aws we can navigate to the iam console that's the identity and access management console and this is what aws uses to allow and restrict access to any service within aws so the first thing we need to do is go to policies and create a new policy and a policy is a set of rules that we can create that can allow or disallow certain behaviors on different services within aws so we're going to create a policy that allows something to add a file read a file and maybe delete a file from this specific s3 bucket and that's all it's allowed to do so in services here we're going to choose s3 because the actions are going to be applied to the s3 bucket then in this list of actions there's lots of things you can do to an s3 bucket uh underneath right we really only care about deleting an object that's deleting an image from a bucket later on if we need to putting an object that's uploading a file to the s3 bucket and then in the read list we want to be able to where is it there we go get an object and we could be selecting more actions here but that's really all the web app needs to do in this case add a file get a file maybe delete your file so that's all i care about there and then in this resources section right here this is where we get to actually specify the individual bucket that we just created so if i click here to add an arn it wants me to add the name of the bucket that we want to apply this to and that's the bucket that we just created if we don't apply this this role will apply to any s3 bucket and that can be bad because you don't want this app having access to any other bucket other than this one that we created specifically for this app so we're only going to allow it to do these things within the bucket but when it comes to objects within the bucket this is files it can upload any file it can delete any file and it can read any file within the bucket as long as it's in that single bucket so we'll add that to the resources and that's it we can move on to next adding tags which i'm not going to add any next review and then we'll give this a name insta app uh bucket policy not the best at naming that's fine create the policy so we finish making the policy that specifies exactly what our application can do but now we need to assign this policy to a user and the user will represent our application so a user can be a user like myself like i can log into aws and i can access the bucket or we can apply a user to an application like the express app and it's kind of like the express app is logging into the aws account only has access to the s3 bucket in the way we applied in that policy and then it can just access the bucket as it needs to so we're going to create a user for the express app so i guess i'll call this insta amazing app that's actually probably good enough and then down here because it's an app it's not a real human user that's going to log in we're going to give it programmatic access next we can go to permissions and here's where we attach the existing policy that we just created so if we go to this little tab here and then search for the policy we just made i've forgotten what i called it insta something oh there it is insta app bucket policy select that and we can add as many policies as we want so i could create rules that allow this app to connect to a database managed by aws or some other services right now i only care about s3 uh we'll go to add tags again not going to add any tags review yep that's good and then we create the user so super easy just create the user and add that policy and then we get to this page and this matters a lot we need to keep track of this access key right here so i'm going to copy the access key and go back to my m and i'm going to write aws access key and then the second most important piece of this is the secret access key which i'm going to show right now and make sure you immediately copy that paste it into the m-file seek secret key and never share this with anyone don't let anyone see your keys ever this is the ability to log in and modify all of the things within that s3 bucket so you need to keep the secret you need to make sure no one ever sees it and this is the way that the application is actually going to be able to access the bucket so now that we've got those saved i can close this page and i am going to leave this alone for a while because that's all we really need so now that we have that data all we need to do is write the javascript code that actually pushes that image to the s3 bucket and to do that we're going to use the aws sdk node so if we scroll down to the mpm page for the aws sdk we'll get instructions on how to install it but there aren't many docs here it's really just kind of like a readme page but amazon does provide a lot of documentation it's just not known for being the best documentation and the easiest to read and even existing in this case so i'm just gonna i'm just gonna install it and then show you how to do this so first thing i need to do is go to my server npm install aws sdk and i'm only going to be using the s3 part of this sdk so i'm going to go into vs code and in my server i'm going to make a new file i think i'm just going to call this s3.js and in here i'm going to require require just the s3 part of the actual aws sdk client and then from here i'm going to need to do two things i'm going to need a function that uploads a file to s3 and another function that basically downloads a file from s3 so when a user uploads a photo it's going to go up to the s3 bucket and then when a user tries to get a photo probably by putting a url in an image tag the server will have to grab it from s3 and then pipe it back down to the client but before we can do either of those things we need to create a new instance of the s3 class here and this is where we need to provide all of the details that we have in the environment variables file so i'm actually just going to copy this and create some variables here so i'm getting the information from the dot m file and then once this is on a server i'll actually have to have the environment variables on that server and then we initialize a new instance of this s3 object using the region the access key and the secret access key and as long as all of those are correct we'll be able to access that bucket uh there's just one more thing i need to be able to do because i'm using a dot m file i need to install dot m the library and then i need to uh require dot m and config it and now good um now this all should work so the next thing i need to do is create that function that actually uploads the file to s3 [Music] so here's the function i've called it upload file uh we're going to pass in that file object that came from malta that has the file path on the server and that file name that's just that unique key so i'm first going to create a read stream using the fs library so i imported that and through that read stream we can actually create this object that contains the read stream as the body the bucket name as the bucket and the key this is going to be the name of the file within the s3 bucket i'm just going to use the file name that malta created then if we pass this to the s3 dot upload function it will upload it to the s3 bucket and then we can call dot promise on that function to return a promise rather than having to use callback functions so a pretty small function to actually get this to upload 2s3 and now that i have this implemented if i go back to my server i can just grab that upload file function from s3 perfect and then in here i'm going to make this an async function and right here we can await upload file and pass in that file from malta so multiple it in the uploads directory uh we send that to s3 we wait for that to actually be successful we should probably do some error handling here but i'm just going to leave it for now and then we'll just send back um which i'm not sure what we sent back let's first actually grab the result yeah the result from s3 and i'm going to console log the result so we should see this make its way to the s3 bucket and then we'll console.log what s3 gives back to us so we can actually see what's going on and the end result here should be that we send enough information back to the client so that the client can then make a request to get the image and then we'll be able to see the image appear even though it's hosted in the s3 bucket uh so let's try this now let's go back to oh now i've done something wrong i can't export because i'm not using modules i need there we go i already did the exports perfect okay so now back on the client i've already got my image selected i'm just going to clear the network log and if we submit this it's uploading the file let's see oh and it came back as a successful request there we go we got that little okay hand uh so it looks like this is the stuff from malta this is just the image upload initially to the server and then this is the information we actually get back from s3 so this is a url that could be accessed to actually get the image data and if we wanted to we could modify the s3 bucket to make these files publicly available and that way we could just use this url and every time someone wants to get an image they don't have to go through our server first they can just go and get it straight from s3 and this is fine a lot of people like to do this i prefer to go through the server though because with most applications i want users to be logged in before they can actually see an image i just want to make them publicly available so instead we're going to use this key right here either one uppercase or lowercase and we're going to use this to actually be able to get the images back one more thing we should do is actually go back to the s3 bucket and make sure that the file actually exists in the bucket make sure we can see it there so i'm going to go to my insta amazing apstagram and it's loading the objects and there it is there's that key for the file that's been uploaded and if i check terminal it matches the key that we actually got back from s3 so that's perfect it seems to be uploading straight to the s3 bucket and then on the way back down so in the server response here what i'm going to do is i'm going to provide an object that just says image image path maybe and i'm going to say that it exists at slash images slash uh result dot key so this is the key this is the file name within s3 and by doing this i'm basically telling the client if you make a get request to slash images slash the the file key i will go to s3 and grab the image for you and then you could just dump this into an image tag and it should work so i'm going to just make this on the server side app.get slash images slash key browse wreck and then i can get the key through the params and then all i need to be able to do is grab that image right this is the key this is the name this is how it exists in s3 i just need to tell s3 hey give me that specific image and i'm going to send it straight back down to the client so let's make that function now [Music] so again this is a really small function uh we're passing in that file key we're gonna hand that over to s3 with the bucket name and tell it that we want a read stream that it will then return to us and then we can return out of this function and the great thing about using a read stream is that if i bring this function into here and i call this to create a read stream then we can just pipe this read stream straight into the res object and that will just send the image data straight to the client so this is a super handy thing about using express so i'm just gonna cheat here a little bit and i'm gonna go and i'm gonna grab this key right here and if my server works i should be able to go into any html client and create an image tag image source and as long as it starts with images because that's the base of my url that i've specified here and then it's followed by the key so images slash the file key that exists in s3 this should make a request to the server the server should make a request to s3 and then pipe the image stream straight back to the client so this should show that image that first image that i uploaded so if i go back to the react app nothing is showing up and my server is crashing cannot read property key of undefined so i did something wrong in the server um wreck.prams.key was that the error message cannot read on server.js okay is it not prams uh right params is not the thing are you sure [Music] so i made a dumb mistake i put wreck instead of res and yeah that was dumb javascript everyone doesn't care which order you put function parameters in anyway my fault not javascripts so refresh there we go okay it's getting it from the server so uh if we actually take a look i'm gonna clear my network quests and go here i can see the image is being downloaded from the server that's localhost 3000 that's actually proxying to low close to 8080 slash images slash the key of the image that goes to my express server which then goes to the s3 bucket grabs the image and then displays it here so that's it and if we take a look at the code we can see we post an image it comes to the server it then gets sent to s3 through this method here then when we want to get that image we can just uh create an endpoint like this where we can grab the key create a read stream and send it back down to the client so this is fully functional it's uploading the image to s3 getting it back from s3 there is still something that i would change here though because the uploads directory is going to keep storing all of these images in this folder and this is unnecessary because once it's on the s3 bucket we don't need it on the server anymore so once this is successful right here on line 21 once we upload the file to s3 it would be great to just delete the file from the uploads directory so i'm going to just include the fs library require and the utils because i want to create an unlink file function that uses a promise instead of a callback function um done this wrong that needs to be util actually i think it is and then i go util dot promisify fs.unlink cool um so this just makes it easy after we've uploaded to s3 i could say await unlink file and it'll be the file path i think file.file all right file.path uh let me just double check that um malta gives us path yeah so path from malta gives us exactly where it's located on the server so we'll just use that path to delete the file once it's been uploaded to s3 and that will clean it up so i'm gonna delete that uploads directory and if this is successful uh it'll create the uploads directory it'll put a file into it upload to s3 and then actually delete the file from the server because it exists in s3 so i'll just test this out quickly let's upload a new file here submit that so if that was successful it still exists in my uploads directory oh no it doesn't okay so yeah that worked uh it took a little while to get to s3 and then back but it no longer exists so that unlinks it there and there is actually another library called uh malta s3 i think that does this in one step so instead of uploading the file to the server first and then uploading to s3 uh it just takes the file as it's coming in through an upload stream and pipes it straight into s3 so it never actually gets stored on the server itself and this can be really handy if you just want to send it straight to the s3 bucket however if you do it this way this does allow you a little bit of time from the file getting to the server and you sending it to s3 so you could perform some actions like maybe apply a filter maybe resize the image maybe resize it into a bunch of different sizes and store them all in s3 bucket if you wanted to and then by having it come through the server on the way back down to the client you can do some sort of authorization or any other things you want in your javascript code before the image gets from the bucket to the actual react client in this case that's it for this video check the description for all the code examples and any other instructions that i'll put down there and check stack overflow where my code examples don't work for [Music] i got you to get it talking on me so my car's attended dancing with the devil i don't bargain with it

Channel: Sam Meech-Ward

Views: 38,286

Keywords: aws, s3, node.js, express, Upload, Images, node js, s3 tutorial, s3 bucket, web development, amazon s3 image storage, express image upload, multer image upload nodejs, file upload nodejs express

Id: NZElg91l_ms

Length: 26min 1sec (1561 seconds)

Published: Mon Mar 01 2021