NodeJs How to upload Files + uploading to AWS S3 using Express, Multer

what's going on guys so in this video we're going to take a look at how we can handle uploading files to your server using node.js and more specifically we're actually going to set up a api endpoint using express that's going to handle all of the uploading of files using a library called malter and in this video we're going to specifically focus on just uploading it to the server and storing it on the server in the next video we'll actually take a look at how we can upload it straight to aws s3 so definitely uh stay tuned for that video as well um but we're going to start off with the basics in this video and then work our way up so the first thing that we're going to do is let's go ahead and create our package.json file so i'm just going to do an npm init y and then i'm going to install express once we have our express installed i'm going to create a new file i'll just call it index.js and we're going to import express then we can create our express instance and then we're going to create our single endpoint that we're going to use for uploading files so i'm going to do app.post so this is going to be a post request and you can set up you know whatever route that you want i'm just going to call the endpoint upload and then here we can hand set up our controller so this is just going to be rec res and for now i'm just going to send back a json that's going to say status all right and so that's all we're going to do for now and then we'll set up our express server to listen on a specific port so i'll do app.listen port 4000 and then i'll just set up the callback just a console.log as usual all right so we've got our express application set up now uh the next thing that we want to do is we want to install the multi library so that's the library that's going to handle uh the uploading of files and i definitely recommend you pull up the documentation so you can either take a look at the github page or even on the express page as well so all we have to do is just npm install multer and let me make this a little bigger there we go so let's do npm install multer once we have multer installed let's import it so we'll do a const multer equals requirement and we're going to start off with the simplest example and we're just going to handle uploading a single file at once so malter is going to come with a middleware that we can use right and this middleware is nothing special it's just a regular express middleware that we can just kind of chain in wherever we want in our route handler so that we can handle the uploading of file so to set up the middleware we're going to call multur and then here we're going to pass in just one option and there's a couple of different options if you want to take a look at it you can actually hit control on vs code and then select this and it's going to take us to the uh to the types uh for multi and it's going to show you the different options uh if you want to take a look at that but we're not going to focus too much on that i'll actually walk you through exactly what you need so here we're going to pass in an object and we're going to pass in just one option which is dest so dest is going to be what folder do you want to upload the files to on your server and so i'm going to create a new folder and by the way i don't think you actually need to create the folder ahead of time multi if it sees that the folder doesn't exist i think it should create it for you automatically but i just want to make sure that you guys understand that this is just specifying uh the folder that you want to upload to so i'm going to create a folder you can call it whatever you want i'm just going to call it uploads and so here i'm just going to pass in the name of the folder or the path to the folder that i want to upload to so it's going to be uploads and then we want to store this in a variable and so i'm just going to call this upload and upload is going to be the middleware that's going to handle uploading the file so anytime someone uploads a file our middleware is going to run and then it's going to upload it to the uploads folder that we just created and so like i said this is just like any other middleware we just do upload and from upload we have to pass in one method so depending on how you want to set it up since i said we're going to handle uploading just a single file you do upload.single okay and we have to pass in one argument all right and that's going to be the field name so when the front end sends us the um the file what is the name of the field that the file is going to be attached to once again you can pick whatever you want i'm going to use the name of file right however you can call it whatever you want let's say you're handling the uploading of an image you could call it image just make sure that when the front end sends it it's going to send it with that specific field name so i'm just going to call this file and then make sure you get that extra comma and that is all we have to do to set up uh multer to handle uploading files so let's test this out and i'm going to do this with postman so that i can generate a request you can do it with your front end uh you can do it with a you know just basic javascript you can do with react doesn't really matter but i'm going to send a request to localhost 4000 that's the port that express is listening on and then the end point which is slash upload so upload and this is going to be a post request now the most important thing to remember is normally you would go under body uh when you set up the body and then you would do you know raw and then json however for multer we actually have to use form dash data so whenever you're sending your front end to send data back to the back end and you're uploading a file and you're doing and you're using multi you're going to have to always send it as form data that's an important thing to remember all right and then here if we specify the key and write the key is going to be whatever you put here so since our key is file we're going to call this bio and then we're going to change this to be a file type okay and then we'll select the files that we want to upload and i'm going to go into this pix folder that i created and we're going to upload one of the files so i'm going to upload monkey.jpg all right and we got that so let's then send this and it looks like it hang so let's take a look at what happened and i realized i forgot to actually start my express server so silly me so actually i'm going to install nodemon and i'm going to go to my package.json and we're going to set up a script start script so we'll do start i'll just do nodemon index.js and let's do an npm start all right so now it's listening up for 4000 let's try this again all right we got a status success let's see if our file actually successfully loaded so if i go to our uploads folder you can see that we do have a file now if i take a look at this um you know express is going to show me the image because it doesn't have the right extension but that's okay uh so you'll see that multi actually gives it a random you know name or id it doesn't take it from the original file name and so we have successfully uploaded a file using the multiplayer library so you can see it's actually very simple to set this up but we're going to keep continuing further and the next thing that i want to show you is handling multiple files how do we upload more than one file because right now this only accepts one and we can actually test this out right if i go to my postman and select the files again and we select two files and we upload this all right you can see that it throws an error an unexpected field so multi doesn't know what to do with this because we selected upload.single so it's going to throw an error no worries not a problem we'll take a look at this in the next part of the video all right so let's set this up for um uploading multiple files and what i'm going to do is i'm going to actually comment this out for now so that you guys have this when i uh upload this to the github page and i'm just going to copy this because for the most part it's going to be almost identical with just a couple changes i'm going to just in the notes i'll say this is for a single file upload and this one's going to be for multiple file uploads all right and i'm going to uncomment this and we can set up the middleware exactly the same way we just specify the destination folder but we can change this from single to and actually let's just actually delete all this and let's see what vs code auto completes for us so we've got single we've got none we've got array in fields so um i think you can probably guess we're going to have to use the array because we want an array of images or or an array of files so we do array and then this time once again you just pass in the name of the field so i'm just going to use the same name which is going to be file and i'm going to save this and now we've got the two files selected once again so let's try sending this and it seems to work and if you take a look at our upful uploads folder we now have three total files because we get the first one that we originally uploaded and then we've got the two other ones so that's how you handle uploading multiple files it's actually really simple as well now if you do want to put in a constraint on the maximum number of files a user can upload because you don't want to allow them to try and upload like 100 files at once you can actually put in a second option and this second option is going to be the max number of files that you want to allow so if you select two we can do that i'm going to actually delete these files for now just so it doesn't clutter this up all right and if i i have two files selected so it should be within the maximum that we're allowing we hit send you can see success both of them uploaded now if i try to do this with three files i'll select three and if i hit send you can see that it errors out and obviously we're not handling the errors gracefully we will take a look at how to handle this a little bit better later on in this video but for now i just wanted to show you how to set a maximum constraint on the number of files a user can upload all right and so that covers the section for handling multiple file uploads then now the next thing that i want to cover is how to um specify multiple different fields right so maybe you want one field that's going to be called um you know profile pic or avatar that's going to be so the user can upload a avatar image and then we might have another one for like an image of their resume or something right so that that's an option that's a possibility maybe your front end needs to uh provide multiple different images to the back end and we want to make sure that the files are associated with the right key so how do we do that and so once again i'm going to do the same thing i'm going to copy this paste it down here and i'm going to comment this out and this is going to be i'm going to just label this as multiple fields upload and this time we don't actually need this line right here or well we do need it but i'm going to rename it so i'm going to call this how about multi-upload actually sorry i made a mistake undo that keep this here but we're going to create a new variable called multi-upload and i'm going to say this is equal to and then we're going to reference this upload middleware we'll say upload and then we can specify fields and then here we can pass an array of the different fields that we want so before right we were just saying uploads.array and then one field so here we can pass an object that's going to represent one field so the first one right we can give the field name whatever we want and i'll say maybe this is uh this is for the user's avatar all right and we can also specify a max count so the maximum number of files for this field i'm just going to say one for now and then the second field will say this is for the user uploading their resume and so i can give this a name of resume and once again we can provide a max count of one or whatever you like doesn't really matter and then here i can just remove this and just call multi-upload we'll restart that and so now it's going to look for two fields one called avatar one called resume so we need to actually set that up so this is going to be avatar and this is going to be a resume and this is going to be a type file and i'm going to remove this now let's select the avatar now so we'll select the avatar pick and then this one's going to be for the resume and we'll select the resume pick and we'll upload and you can see that we were successfully able to upload all right and both of those files got uploaded so that's how we set up multiple fields you could set up as many as you want i just wanted to show you at least one example where we have more than one field now now that we have a basic understanding of how to use malta i want to do a little bit more of a deep dive and take a look at how it's kind of working so one of the cool things is that because this is a middleware what the multi-middleware is going to do whether you use multi-upload uh whether you use the upload.array or single it doesn't really matter it's going to be the same exact behavior but it's going to take all the files and attach it to the request object under a property called files so i can do a console.log rec.files right and what i'm going to do is i'm going to do this upload again and we're going to take a look at what it prints out all right and so you can see that it printed out something so what happened here well it looks like it prints out a couple of things so we see the field name and then we see all of the files printed in it or all the files uploaded for that specific field so it's an array we only did one file for the avatar field and you could see a couple of things right we could see what's the field name we could see what's the original file name uh we can see the mime type this is important this is uh malta will actually figure out what type of file it is that's and that's a nice feature because you know you don't want to trust the uh the extension of the file because i could take a text file and just call it dot png but it's not actually a image it's a text file so you don't ever want to trust the name of the file anybody can change that instead you want to use the mime type from alter to figure out what's the type of the file when we can see the destination we can see the file name and then we can even see the path to the folder was uploaded to so this is going to be the folder and then the name of the file i think it's double backslash for me because i'm on a windows machine but it'd be a regular path if you're on linux or mac and then we can also see the size of the file in bytes i assume and then we see the same thing for the resume as well so this is a cool little feature so um you know anytime you need to have this information you've got it at your disposal under rec.files okay so the next thing that i want to do is you can see that multi automatically generates some random name and if you want to customize the name it's going to require a little bit more work so i'm going to show you guys how we can uh customize the name so that we can you know use uh whatever naming pattern that we want for our files and for this video we're actually going to make use of the uuid library to generate a unique name or a unique id and then prepend that to the original file name so actually what i want to do is if we take a look at you know the original name right now it just says resume.jpg so when i want to upload this i want to give it a unique id then dash then resume.jpg so that's the format i want to use obviously you can use any naming convention you want i'm just going to show you how to do this just one time and you'll see that you'll be able to customize it however you want so i'm going to comment this out once again and i'm actually going to grab the multiple file uploads for now because it's going to be the same thing here but we're just going to customize it and let me uncomment this and this is going to be custom file name all right and what we're going to do is we're going to call the multer object that we imported and there's going to be a couple of different options so by default multer supports uploading files to disk and uploading to memory so if you just want to start the files in memory and obviously you can um you know install other packages that also allow multer to upload to s3 and a few other places in a few other object storage places as well um but these two are built in so i'm going to show you since we are working with this stores which is just uploading it to the server we're going to call the disk storage method and here we're going to pass in an object and the first object is going to be the destination so we get to customize where we want to send the files to and you have to pass in a function that gets the request the file and a callback function and here to specify where you want to send the files to which is the equivalent of you know doing this you just call the callback the first parameter is going to be null then the second one is going to be the path to the folder so this is going to be uploads you can put the slash you don't have to put the slash it doesn't really matter now the next thing we want to do is customize the file name so we can call file name and then here you do whoops wreck file callback all right and then to specify what you want the file name to be you just do callback once again first parameter is going to be null then the second one is going to be what the name of the file should be so this is going to actually run this function's going to run every time we upload a file so we what we want to do is i want to give it a pattern that's going to look like this where we do where we call a uuid you know and then it's going to generate a unique identifier and then we do that and then we grab the original name all right very simple so how are we going to do that well let's install the uuid package i think it's just uuid all right and we can import that now and so i'll do const uuid equals require uuid and we'll do dot v four okay so this is going to allow us to generate a unique identifier and here uh we're going to destructure out from the file right this file object is actually going to be um basically what we got printed out here it's going to have all of these properties so we'll get the original field we'll get the field name the original file name which is what we ultimately want and then any other data that you that you want to use so i'm going to say const original name and then we'll just uh destruction out from the file object all right and so once we have that we can then create a template literal and here i'm going to say we're going to generate a unique identifier using the uuid library so i'll do uuid and then we'll do a dash and then we can once again pass in original name and i think that should give us everything that we want uh and so now we have our multer object and we actually want to save this so i'm just going to save this and call this storage and here instead of just passing a simple object with the destination folder we're going to pass in something else we're going to pass in this new storage object and if you actually want to see what options you have just do that and then just see if you hover over it with a perhaps you could see the multi options which if you want to you can actually take a look at all that but we just pass in storage and that's all you have to do and then we can call upload dot array once again and i'll ignore the file limits so now i'm just call it file all right so let's test this out and now everything's going to be uploaded under a field called file so i will we don't need the resume or the avatar we're going to call this file and let's upload more than one file at a time so i'll select these two hit open hit send all right both of them uploaded what's most important though is let's take a look at the name of the files and let's see if it did exactly what we want so you can see here we've got a uuid and then dash resume.jpg so that was the original file name and then we've got this one uuid dash and then the original file name which is monkey.jpg pretty cool so that's how you change or alter the file name before it gets uploaded so you could use whatever naming convention that you want okay so the next thing i want to do is i want to show you guys how you can make it so that users can only upload a specific type of file right you don't want users to be able to upload any old file it could break your application if you want a user to upload an image we need to make sure that our code handles the logic so that we force the user to only upload an image if we expect a pdf then we want to make sure that we only get a pdf if we expect a video then we want to make sure that we get a video we don't want to spend time processing a file that we can't even accept anyways so how do we do that well like i said uh when we have access to the file object we can actually see the mime type and that is going to tell us what type of file it is and then we can just run a simple check and just say hey if mime type equals image jpeg if we're looking for a jpeg you just set it equal to this string and that's it it's a simple if statement and you'll see that the the uh format of the mime type is always going to be like the parent type so like uh this is the image and then you'll see all of the different images types so you if it's a png it'll be image slash png or whatever other file type if it's a video would be video slash you know mpv4 or or video slash uh what's another video type wav so if you want to say um you specifically want a jpeg you would do image flash jpeg if you want all images then you would just say if the if the name before the slash is an image that's the one i want then that's a good enough check so that's what we're going to do we're going to say the user can upload any type of image but we can't accept any other type of file but we can't accept any other type of file so let's set this up and so this is what's referred to as a file filter and if you actually want to see the documentation i think it's here if i search for file filter there you go and let's see if we can find an example of that so a file filter it's going to be a function that gets the request file a callback and it's very simple you just you take the callback and if you want to accept the file you just say null and then true that means we accept the file that means it's of the proper type and if you want to reject the file because it's an incorrect type you do null and then false and if you want you can even throw a specific error as well so let's do that and here i'm going to create a new function called file filter and we're going to use the same properties the document show so request file callback all right and so with the file object it's going to be this exact object so we just want to grab the mime type property so i'll say if file dot mime type right and if we wanted to say i wanted to allow just jpeg then i would say equals equals equals image jpeg however we want to also accept uh pngs and any other image file format so how do we do that well we're going to split this string up and we're going to split it at the slash so it's going to create an array with the first entity being the first um the the text before the slash and then the second um item in the array is going to be the text after and so we can just say if the first item in the array first element in the array a equals equals image then we know it is a image type file so we'll say file.mime type dot split so this is going to give us that array we're going to split on the slash so we'll have but what that's going to look like is it's going to say it's going to look like this you'll have an array that says image i mean technically it'll be in quotes and then you'll have jpeg okay so we just want to grab the first item in that array and so i'll do 0 and if that equals equals just image that means it is of the proper type so to pass or say we want to accept this file you do call back null and then true else i can say callback equals null false and we can also throw an optional error so i'll say maybe new error and we could say file is is not of the correct type and then once we have the file filter i'll go into here where it says upload and then pass in another property called file filter and that's all we got to do so let's test this out so i'm going to upload this well actually i don't remember what i selected um but this time i'm going to upload a doc file and this should error out our application all right so you can see it technically did crash our server like i said we haven't handled uh gracefully handling all of the errors yet but that's okay but we can see error file is not at the correct type perfect let's restart our app and let's try it with an image file select files and uh we've got a jpg file and if i upload this you can see status success and we can see that uh not that's the monkey but uh where's the dashfall.jpg got successfully uploaded all right now the the last thing that i want to show you guys before we actually figure out how to handle errors better using um you know express's built-in error handling middleware is i want to show you how we can add more limits right so more constraints on what the user can upload because we did put in a check to see that hey are they uploading the right type of files but there's other things as well right we also want to put in limits on what's the file size right we don't want to be able to allow a user to upload a 40 gigabyte file that would be a waste of our resources especially if we just expect them to upload a simple image that shouldn't be 40 gigabytes so how do we do that well i want to take a look at the documentation and um look for limits you'll see the different options that we have all right so you can see here this is the limits key that we can provide and let's see if uh here we go these are the different limits so we can specify the field name size which i mean i don't really cares about that the field size don't really care um the max number of non-file fields that doesn't really matter here we go we got file size here we could specify what is the maximum size of the file you can also specify what's the maximum number of files the maximum number of parts per file the main thing i want to focus on is file size and the number of files so let's set this up and so under file filter we can just do limits and then we pass an object and here we can specify what is the file size and so this is going to be in bytes and so this would be one byte if you wanted to do one kilobyte that would be a thousand of these and if you want to do one megabyte that's a thousand of those so six zeros i think should be one megabyte okay and so if i save this and now we try to upload a file that's uh i think this should be bigger than one megabyte let's see what happens all right it throws an error and it says multi-error file too large awesome so now we've set up limits for our file and if you also want to specify um you know the maximum number of files you can just say files and then specify a number so i'll say two and so let me grab a couple of well i'll grab three files hit send and once again air it out and it says well i forgot that i have the file limit so let me actually add a couple of zeros so that we can see this one erroring out send too many files awesome all right so now let's set up some better error handling so that uh you know our application doesn't actually crash out and we can send back a nice response to the user so with express we have a error handling middleware so if you do app.use and then specify a callback function with four input arguments the first one is going to be error then we got request response and then next okay so this will run any time we throw an error inside one of our controllers and what we can do is we can actually do a simple check um some altar comes with a specific type of error that it throws right you can see there's a an object called multi-air so this is a an extension of the built-in error class for javascript that multer builds upon so what i can say if error and then we say instance of is an instance of that's basically checking hey is this a multi-error we can actually call the multi-object that we imported and then call dot multi-error so this is going to check to see if this is one of the different multi errors that can be thrown and that's all you have to do so here at this point we can then run some further checks and actually what i recommend you do is hit control and open up the documentation so you can see that the multi-error can be of you can see the different arguments we've got the name code message and in in the error code is what we're ultimately looking for so the different error codes are limit part count file size so this is what's going to get thrown for the error code if the files is too big um unexpected file uh field count so these are all the ones that it supports so we can just do a check to see if it's a specific error code and then send back a specific response so here i'll do like another check and i'll say if error dot code right remember we want to check the code equals equals and then i can say you know limit file size then i can just say res return res.json and then i'll just say oops i'm on an object and i'll just say message file is too large we'll save that and what i'm going to do is i'm going to lower this back down to some smaller number and then i'm going to select a file and this is for sure going to be too big and if i hit send look at that so now we get a nice proper response where it says hey message file is too large and hopefully our application see our application didn't crash because we actually handled this more gracefully alright and we could do a check for all of the other different messages as well and what i'm actually going to do is well actually let's do one more actually under here i'll do if i'll say error.code equals equals limit file count then we'll say i'm just going to copy this statement right here and this time i'll say file limit reached and now if i select three files open it hit send well once again the file limit is also too big so let's add a couple of zeros just to make sure that everything still works the way we wanted to hit send file limit reached awesome and i want to do one last check and i want to change the behavior of our file filter so for our file filter i want to make sure that we use the built-in limit unexpected file for the wrong file type so when we throw an error instead of sending a regular javascript error i'm actually going to throw a new malta dot multierror and the code is going to be and i'm actually just going to copy this string that's the code so next time we send the wrong type of file we're going to throw a multi-error and then down here we could just run another if statement so i'll just copy this and then change this to be what was it again here we go and we'll say file must be of image type file must be might be an image that's better and obviously we can also add in a status you know i'll just say 400 for everything let's try this out now i'll upload a pdf and there we go proper message file must be an image so that's how we can handle errors in our express application for the various different multi errors and so hopefully you guys enjoyed this video in the next one we'll take a look at how we can upload the images or the files to aws s3 uh so that that way um we can take a look at a different method of uploading instead of just uploading it uh to the machine locally okay so now we're going to take a look at how we can upload files directly to aws s3 and that way instead of uploading it directly to our node.js server which you know i think most applications you wouldn't upload it directly to your node.js server instead you'd upload it to some sort of storage service like s3 or azure's equivalent or even gcp's equivalent and that way on top of that you can also set up public access so that if you're building out like a web application your front end can retrieve those files or images from s3 directly without going through your node.js server and so the first thing that we have to do is we have to create a bucket in aws and then we have to create a user that our node.js application is going to use so that it has the proper credentials to connect to aws so that it can actually upload files and retrieve files so let's create a bucket the first thing we have to do is we have to go to s3 so if you don't have the button here just search for s3 and it'll pop it up right there all right and what we want to do is we're going to create a new bucket name it whatever you want but it's got to be a unique name so you can't use the name that i use so i'm going to call this um aws s3 node.js upload bucket and we can select your region and i'm going to use the default region so make sure you remember which region you're using and then everything else can just be left as default for now we will change one more thing a little bit later on so we'll select create bucket and so now we've created this bucket where is it aws s3 node.js upload and you'll see that everything looks good up to this point and so now what we want to do is go ahead and just copy this name because we are going to need it in a few seconds what we want to do now is go into iam so this is where we set up the permission so we want to make sure our node.js application has the permissions to actually access this bucket and we're going to do two things here so the first thing that we need to do is we're going to create a policy that's going to have the exact rules that will allow our application to access just this bucket we don't want to give our application unlimited access to all of our aws account we want to give him just access to things that he needs and nothing more so we'll go to policies and we're going to create a brand new policy all right and then under service we'll choose service and then we want to search for s3 and then here we can select all of the policies that it should have and so a couple of the policies that we'll need we'll do get object so that's going to allow us to get a file after that's already been uploaded so just search for get object so once again that's for retrieving an object that we've uploaded to s3 so we'll need that we'll also need to be able to upload an object to s3 that's going to be called put object so we'll do put object and then we'll also need to be able to delete so let's look for delete object and we've got that as well and then you could also take a look at some of the other ones there's like get attributes what does it get object attributes um you could also include that if you just want to be able to query the attributes of a specific object and then we've got all of our policies now the next thing that we have to do is specify the resource so right now he has uh this policy has access to all of our buckets but we want to just say it should access just the brand new bucket that we created i don't want him to be able to touch any of the other buckets that we that i have so we'll go here and then i'll select arn and then like i said i told you to copy the bucket name so just paste that in there and then for object name i want to basically say for everything in this bucket give me access to that so i'll just do a star so that's kind of like a a catch-all a wild card so i'll select add and now we've set that up and this should be everything that we need to do from a policy perspective so we'll hit next tags and then i don't really care about making a tag and then we'll give this a name so i'll just say this is node.js alter s3 policy example doesn't really matter what you call it just make sure it's uh got a name that you can easily identify all right so now we have successfully created our policy the next thing that we have to do is we have to create a user account that the node.js application is going to log in as essentially so let's go to users you'll see all of my users you probably if you haven't worked with aws you'll just have the default admin and that's okay so we'll just do add users give it some arbitrary name and once again i'm just going to say node.js aws s3 multer example and what we want to do is we want to select access key so this is going to be programmatic access so it's not like i'm going to go to the aws console and type my credentials it's going to be all through access keys all right then we'll go to permissions and here we're going to attach the policy that we just created so that's going to dictate what this user has access to so we'll attach existing policy and i forget what i called it i think there's the word example somewhere in there yep here we go this is the one we want so just search for the name of the policy select it tags don't really care about tags review and we'll create user and it's going to provide us the access keys and the secret access key so make sure you write these down i'm just going to download this because i know i'm going to forget it and make sure you don't share these keys with anyone else because if anyone else gets access then they can upload and delete anything from your s3 bucket all right so we've got uh aws setup and so the next thing that we need to do is let's install the necessary library in our nodejs application and i'll show you how easy it is to upload files uh straight to aws s3 all right so let's set up the aws sdk in our node.js application and there's one thing that i want you to know before we get started there's actually two different versions well technically there's three versions uh however i don't think anyone really uses version one anymore so there's version two and then there's the i guess it's fairly new version three and i wasn't really sure which one to cover because i think a lot of people are still using version two so i was like hey let's just go ahead and take a look at how to configure both of them so we're gonna start off on version two and then i'll also show you how to do this with version three it's a little bit different i kind of like the way version two is set up but um you know i just wanna make sure that you guys can do both anyway so we'll start off version two then move on to version three so to uh set up version two we all we have to do is just do an npm install aws sdk and then to import it well we don't need to import all of that nonsense we just do aws sdk so let's do that now let's do npm install aws sdk so this is gonna be version two all right so now that it's installed um i can put all the logic inside the controller but uh instead of that i'm actually going to create a separate file so that we can kind of have all of our aws uh code kind of contained to one single file so that we don't have to look for multiple places in our code if we ever want to change anything with regards to which uh storage servers that we're using or maybe we want to upgrade our uh aws sdk version and i don't want to have to look through all of my code so i think it's easier just to set it up so that it's in a separate file so i'm going to create a new file and i'm just going to call this s3 service dot js all right and since we're using the s3 service i'm going to import uh the aws sdk and we're going to destructure out just the s3 service so i'll do const and then s3 capital s so don't forget the capital all right and that's going to import the aws sdk so it's going to allow us to handle uh working with s3 i'm going to create a function uh we're going to export it so we'll do exports dot and i'll call this s3 upload v2 because we'll have one for v3 one for v2 and this is going to be an asynchronous function all right and the first thing that we want to do is we want to initialize an s3 instance so that we can actually talk to aws s3 so we'll do const s3 this is going to be a lower s and this is going to equal s3 new s3 okay now if we actually go back to the documentation we actually have to provide the credentials right we have to make sure that we hit provide the access key and the secret access key so the old way was we would actually pass it in as an object in here and we would provide uh what is it key so access key id secret access key id you can see that this is deprecated so what aws wants you to do is use environment variables right that's kind of a safer method they want to make sure you don't accidentally hard code your credentials into your code because then if you upload it to github then everyone has access to your keys so we're going to use environment variables so um you know you can use whatever library that you use to set environment variables or you could just set it directly on your machine i'm going to use dot nv so we're going to import that so i'll do npm install dot nv and if i go back to my index.js at the top of this file i'm going to make sure to import dot envy so i'll say require gotten the dot config and then let's create a dot env file and let's figure out what are the environment variables that we need if you don't know if you're wondering where i got it from we can just take a look at the documentation and it looks like that is not here okay and getting started using setting up sdk for javascript configuring sdk this is probably where we want so getting your credentials setting credentials so you can see here setting credentials on node.js from environment variables here we go so these are the two main ones that we need aws access key id aws secret access key so we'll do aws underscore access underscore key underscore id equals whatever and we'll get that later and we have aws underscore secret underscore access underscore key so that's the x key in the secret key you can also specify your region if you want so mine is us east dash one so hopefully you wrote down where you created that bucket and then we're also going to have an extra one for the bucket name so i'll do aws underscore and this one you can call anything you want underscore bucket underscore name all right and let's fill in all the values so i got my excel sheet that i downloaded so this is the username that doesn't matter we want the access key this is the access key i'm going to copy this paste it in it should be an equal sign and then let's get the secret key and then the bucket name let's go back to my web browser it's called aws s3 node.js upload okay so that's going to be all of our environment variables and we can delete this object so right there s3 or our aws sdk is already fully configured so it should have access to s3 already now the way to actually upload a file is we're going to call uh s3 and we're going to call a method called upload pretty straightforward right and here we have to provide a parameter object with uh with our configuration so let me show you what this looks like it's just an object and i'm going to call this param and there's going to be a couple of different options we have to provide uh what's the bucket we want to upload this to so it's going to be bucket with a capital b and because we set our bucket in environment variable i can just do process dot env dot aws underscore bucket underscore name the next thing we need is the key so this is going to be oops there you go uh this is going to be the name of the file when we upload it and i'm actually going to just do the same thing that we did for the other files where i set the uuid dash and then the original name so let's actually try that and i'm going to import the uuid library as well from here so let me copy the import statement and we can also specify what subfolder within our bucket that we want to upload to so if right now if i just do you know name of file or whatever it is it's going to upload it directly into the bucket but if i want to create a folder and then put it in that folder i can always do whatever the folder name is in this case it'll be uploads and then slash and it's going to put all of the uploaded files into a folder called uploads and i think that's going to work a little bit better for us and that's what i'm going to do and then here i'm going to do uh we're going to do the uuid and then we have to give it the original name so for now i'm just going to write the word name just hard-coded it's okay because we don't have access to the actual name then the last thing that we have to do is we have to provide uh the body property so what is the body uh it's a it's actually a buffer object so this is going to be our file in memory so node.js is going to actually take the file from the client as a in memory and it's going to be stored as a buffer and then we provide that buffer to aws so it's going to be you know some buffer and what's going to happen is the way i'm going to set this up is we're going to receive the file object from malter right and if you remember the file object has a couple of different properties let me see if i can scroll up and show you guys here we go so this is a file object right we have access to field name original name encoding mime type destination all of these properties and when we change the configuration so that we upload it to uh malta using the memory storage we'll also have access to the buffer for the body so i can just call in file dot buffer and for the name i can actually get that from file now so it's going to be file dot and it's going to be this name right here original name and that should not be capitalized do not capitalize it i don't know why they didn't capitalize the word name in there but that's all right and once we have the parameters we just provide the parameter here i think by default s3 is callback based but you can turn it into a promise by doing promise which is simple enough and then here we can just do a const you know we can do a const result equals a weight and then we can return result however i'm just going to trim that a bit and just return this directly okay and that's going to be our upload function for now we're going to tweak it a little bit because uh eventually we want to be able to set this up so we can upload multiple files at once right now it only takes one file but that's okay let's keep things nice and simple now from our previous example if we take a look at our uh configuration right we had uh this uh we called the malta object and you can see we used a the type of storage right that's the multer disk storage right that's what stores it onto the disk of our node.js application but we're going to use our memory storage which also comes built in with a multer so instead of storing it physically on the drive it's just going to keep it in memory because we're going to immediately as soon as we receive it upload it up to aws and so we no longer need storage here and i'm going to create a brand new storage and i'll say cons storage equals got memory storage and that's all we have to do so i'm just going to take storage and it's automatically going to get pulled into here so we're just setting that to this new storage object instead of the old disk storage where is it this is the disk storage this time we're just calling multer.memory storage that's going to store the file in memory and then we're going to pass it into upload here so everything else we can keep exactly the same actually so that's all we have to do we can remove the limits for now if you want or you can keep it in there it doesn't really matter and we're going to still use the same file filter so that we only get images and then within our controller i'm going to import the s3 upload function and so now this is going to be an asynchronous function and i'm going to do a const result equals await and we're going to pass in rec.files well actually we're going to get multiple files so i'm actually going to extract one file just for demonstration purposes so i'll do file equals rec.files and then we'll just grab the first one just keep it simple and then we'll send file over and if you want to we can also update the body and say this is a result just send the result doesn't really matter okay and we'll have to start this again all right so let's try this out let me grab my files and i'm going to select these two files open that and let's send it and it looks like we got an error and it says parm is not defined okay so i definitely misspelled something this should be param okay hopefully just just that no other issues all right and it looks like everything worked and let's take a look at the result so the result that came back interesting so it's got a couple of different properties so you can see we got the e tag we've got the location of the file right so this is if you ever want to access the file this is the actual url and then we've got the key so the key was upload slash and then we've got the uuid and then the original file name and for some reason it gave the key twice not really sure why but that's all right and then we've got the bucket it was uploaded to so it gave back all of the data that we need we don't actually need to send the result back to the to the client but you could see that uploading files was pretty easy but let's just double check and make sure that it's actually in the bucket so i'm going to refresh and you can see that it did in fact create a folder so let's go into our folder and we can see that we do have the resume.jpg and if you click on this we can actually download this file so if i select download and we've got the image now one thing to note is that if i actually try to go to the url oh i didn't mean to click that and then just paste it in here right you see that we get an access denied so this is due to our bucket policy right now it's set up so that public users do not have access to it only people with the specific credentials can access it through you know the api basically so uh you know if you're building a web application you definitely want to make sure that your your client can actually access the images so they can load it on the web page so in the next video we're going to take a look at how we can set up the policy in s3 to make sure that public users do have access to the files and the images okay so one thing i want you guys to do is go to this page right here so you can just copy this url right here and go down to bucket policy example so aws is going to provide us some examples of how to set it up so that public users have access to these files just for read abilities we shouldn't allow them to delete any files we shouldn't allow them to add new files just be able to access the files themselves and so there's this section right here granting read-only permission to an anonymous user which is like a public user or you know your client or front-end it doesn't really matter but we can pretty much copy this exactly and all we have to do is just update the bucket name uh when we paste it in so let's copy this and then under permissions for your bucket go to our bucket permissions where it says block public access we want to change this and uncheck that so we'll save this and we're gonna have to confirm so this is gonna give public access and then we wanna put in a policy by hitting edit here and then pasting this and then all we have to do is just copy the bucket name right there under the resource and this is basically saying any files within this bucket and if you want to you can even be more specific and say under the upload folder grab anything from there but it doesn't really matter how you do it okay and we've got our new policy and so if i refresh this page now you can see that it automatically downloads it and i can now embed this url as a image object inside an html file so that it works like a regular application all right and so now in the next section we're going to uh configure it so that we upload multiple files and it's going to be a little bit trickier but you'll see it's not too bad it's just pure javascript and handling multiple promises at one time okay so going back to our code you can see that right now our function only takes one file and we obviously want to be able to upload multiple files at once so how do we do this well first of all we're going to i'm going to just change the name just to be files just to represent we're going to receive more than one file and under index.js where we go to our controller and we're passing just one file we're going to remove that and i'm going to pass all of rec.files so now we're getting all of these files and you have to understand we're getting an array of file objects so how exactly do we upload multiple files now so there's no like special method from s3 that allows us to upload multiple files or bulk update or bulk upload all at once and said we have to call this method multiple times and you don't ever want to just do something like you know i'm sure you guys are thinking like hey let's just do this and then we define a separate parameter and then we can provide that with param2 param3 and so on right that wouldn't work because then we would send one file wait for that to finish send the next file wait for that to finish send one file so we're doing it all synchronously and we don't ever want to do that we want to do it all asynchronously so we want to fire off all three of these all at once so how exactly do we do that it's pretty simple we have to make use of a special thing called promise.all so before we even do that what i want to do is i want to generate the params for all of the files so remember we're getting an array and what i want to do is i want to iterate over that array using the map method so i'm going to say const params this is going to return an array of parameters which equals and i'm going to say files dot map and so we're going to iterate over the file and here we're going to get access to each individual file within the array of files and remember like i said each individual file is going to look like this we're going to get all of these fields so we're going to pass this into the callback function and we're going to get access to this file so we're going to do the same exact thing we're going to return an object with the same exact property so i can literally just copy all of this directly and i can remove this so we don't need that so we now params is going to be an array of all of the parameters for each of the different files now the last thing that we need to do is i'm going to do promise dot also this is how we um upload multiple files at once we're going to uh basically await all of our promises all at once and we're going to have to make use of the map method again so i'm going to say params right that's going to be this array of parameters and i'm going to map that and so we'll get an individual param and here we're going to do the same thing we did well i already deleted it but it's going to be the s3 dot upload and then we'll say we're going to pass in the param that we are iterating over and then do a dot promise and then now we can await all of the results and then we can just return results we can just return up here as well it doesn't really matter which one you do actually i'll just return here as well and we don't need that so we'll get that and then the index.js you'll see that we get results now and you'll obviously want to put this in a try catch block in case you know something errors out and if we want to we can do a console.log of results and then i'll just copy this res.json to over here and if there's an error um you know handle it accordingly i'm just gonna do a console.log just just to show but you just handle it like you know any in any other controller if there's an error you know you send back an appropriate message to the user okay and so let's restart that make sure you save all your files and i'm going to send two and it looks like yeah i forgot i accidentally messed up my request let me add a new request i'm going to make sure to select two files and let's try this out and it looks like we got an error and it says result is not defined i'm not sure why it's throwing that error let's take a look oh it's because i still have this right here where i'm sending the result back to the user okay that's simple mistake let's try this again and we get a status success and if you take a look at what we printed out you can see the two resulting objects so we've got the first file which is the resume and then the second file which is the turtle and so that's how easy it is to set up uploading of files to aws using the version 2 sdk in the next section we'll then take a look at how we can do this with the version 3 sdk it's a little bit i don't know i felt like it was a little bit more confusing but it's it's still not too bad it's going to be similar to logic okay guys so for setting up the aws sdk version 3 it's a little bit different right we're not going to be able to do this simple npm install aws sdk instead if you actually take a look at the documentation it's going to be installing it on a per resource or per service basis so if you want to work with dynamo db you would do npm install aws sdk and then client dynamodb if you want to do for s3 it would be client-s3 so it's per service it's a little bit different but let's go ahead and do that now so i'm going to do an npm install and we want to do at aws sdk client dash s3 now remember this package will only allow us to work with s3 we can't work with any other um services within aws we'd have to install the respective package for that now when it comes to credentials for the version 3 sdk it's the same exact uh system so we have the same environment variables we don't have to change anything it uses the same key access key and secret key so outside of that i think we can go and create our um function and so i'm going to do once again in exports dot and i'll call this s3 upload v3 which equals async and i'm going to show you how to do with one file and then we'll take a look at how we can do with multiple files like i said doing it with multiple files is a little bit more confusing i want to make sure you guys understand how to do with one file before we move on so from the sdk we want to import s3 with a capital s client and this is going to be from at aws sdk slash client s3 and within here just like with v2 we have to initialize it first so i'll say const s3 client equals new s3 client so we create an instance and we want to do the same thing it's going to require the same exact parameters so let's create an object called param which equals and actually i could just copy all of this in except for the return we don't need that so same thing we're going to need to provide the bucket the key for the file as well as the um the file buffer and then we're going to call s3 dot client s3 client dot send and here we have to create a new object and specifically we're going to call the put object command so we're going to have to import that so we'll do capital p ut object command and we should let just let vs code import it but if you guys want to see what the import looks like it's just put object command from the sdk so this is how we put an object into the bucket and then here we just pass in param and what i'm going to do is let's go ahead and delete everything from the bucket so that we can verify that we successfully uploaded this file from from the new sdk so let me just delete this and this is always a pain we'll delete the objects and so now our folder should be empty and it is perfect and once again we can just do a return and that's going to uh this is already a promised base method so it's going to return a promise so we could just return it directly or if you want to you could always do you know const result equals and then we can await it here but doesn't really matter i'm just going to return this and then on index.js i'm going to comment out version two and actually let me keep this here and i'm gonna create a second one so that you guys can at least take a look at the difference between the two and i'm gonna just comment that out and then here we're gonna import version three and then we'll let vs code import it double check that it imported it and we've got version three perfect and once again we're going to do just with one file so i'm going to say const file equals rec.file and we'll just grab the first one and then we'll send pile over and let's try this out now so i'm going to send the same two files we got a success back um it looks like there's an issue because it says undefined here and i suspect that's this console.log so let's take a look um i think something probably went wrong and we'll double check here let's see did it upload it yeah it didn't upload anything that's okay so let's see what broke so we call this and i think i just forgot to save it so let me try it again i think there we go now it works so yeah i just forgot to save it so the result of the version 3 sdk is a little bit different you can see it includes the e tag as well but it doesn't really provide much other information so if you get a status code of 200 that's a good thing that means it went through if you got like a 400 a 401 or 403 anything in the 400s or 500s that means something went wrong but 200 means it's good and if we actually go back to the console we should see after refreshing it that we did in fact upload the one image now the one thing i don't like about the aws sdk version 3 is that it doesn't automatically return the url for us so you either have to run another request to get that object's details or you know it's uh we always know the structure of what the url is going to be and you can see that by going here uh and then going to selecting this image and you can see the exact url so uh your url is always going to be this first part uploads and then the name of the file so your backend doesn't even need to retrieve that url from aws it can always build itself you can just store this as an environment variable and then you just append the image name that you created in your code when you passed in the parameters which would be right here so you can just append that and then you can construct the url yourself so something to keep in mind it's a little bit different in version three and so uploading it that's all you have to do for one file let's now convert it to multiple files and it's going to be the same concept we're going to do the promise.all and then map all of them together so i can actually copy this first map statement here to create our params and then this is going to be files and then we're going to do a await promise.all and we can say we're going to iterate over the params or map over it params.map and we're going to get access to an individual param and then here we just call this right here and then we can just return this and we can get rid of this line all right and i think that should be everything that we need we just have to update the controller so that we are passing all of the files so i'll just do rec dot files and let's try this status success that looks good and we did get an output so let's take a look so the first file we can see status cut of 200 so everything's good second one status code of 200 everything's good so we successfully uploaded two files at once and if you want to just double check in your folder we should now have three files in there and so that's how you handle uploading files to s3 so hopefully you guys enjoy this video it's you know uploading files to node.js uploading files to aws3 using node.js dead simple with the multer library and hopefully you guys liked the video if you did please like and subscribe

Channel: Sanjeev Thiyagarajan

Views: 34,685

Keywords: nodejs, express, upload, files, javascript, programming, aws, s3, bucket

Id: jwp4U6v-3h4

Length: 76min 30sec (4590 seconds)

Published: Wed Mar 16 2022