File Uploads with React, Node, and AWS S3

hey it's ryan here and i am building this product course lift this is a course hosting platform for course authors if you are a course author and you want to uh to have some guidance as you're selling your course and marketing your course um in terms of how to find your audience and really get your course out there course lift is aimed to help you do that and right now i'm building out the um the spot where you would upload videos or other files other assets and i wanted to show just some of the wiring that i've got here for uploading the files it's not in a complete state but i figured i'd show what i've done here to get this spot together basically what i do is i can upload a video i've got some placeholder stuff here right now but i've got this real progress bar that's working and the file once it gets uploaded goes over to s3 so i've got this uh this s3 bucket in my aws account and once the file gets uploaded to s3 there's a background job that get gets kicked off to send the video over to muse.ai and muse is a platform that takes video content and does some ai stuff on it and allows you to do things like search through your videos by text and it will transcribe your videos it'll give you the text of your the the spoken word words of your video in in text form um very very cool platform i found it recently and it's a it's a really neat tool that i think will be really valuable for course authors and their students uh the idea is that students will eventually consume the videos that end up here from each course author so i wanted to show some of the wiring in case it's useful i've got this react application and this is a redwood js application i'm using typescript in it i have got a bunch of stuff going on but i wanted to just in particular show what i'm doing here in this drop zone area so i'm using react drop zone i've got the use drop zone hook from it um and what happens here is i've got all these functions that come from the use drop zone hook they get applied to various elements down here like this is my drag area here you apply the get root props function to it and then you can set up your inputs to to take your your drag and drop area uh you can also as you saw you can click on it and and upload files that way and what happens is when a file gets dropped is you can hook into this on drop function and this accepts an array of files which you can then operate on so what i'm doing right now is i'm just operating on the first file that gets dropped there will be support for multiple files at a time but i'm just at this point using uh you know doing support for for a single file what i've got going on here is i take a file and i append it to some form data so i've got the form data object here i am appending a file that that first file in the array and then i'm using axios to post it over to this endpoint that i've got on an express server which i'll show in a second uh passing in that form data as data and then the so the reason i'm using axios here instead of something like fetch is because there's this on upload progress uh callback this is super useful for reporting the amount that has been uploaded so that you can do things with a progress bar this is the easiest way that i found to to use a progress bar kind of thing where you need to get the amount that's been uploaded and this was you know pretty simple to wire up instead of trying to wrangle my own code together to make it happen so uh when i get some progress what i'm doing is i'm setting some state here i've got a file name that goes on state for set file upload i've got the amount completed and the presence of something in state here for file upload is what i'm using to actually show this area down here which is that placeholder that place kit and placeholder image with the progress bar that's what i'm doing using here the the file upload object to be able to show this area so that's the front end of things thing the the file gets uploaded via axios to to an endpoint on my express server and i'll show that here and one note here is redwood helps you to set up serverless functions for your api so you can tap into that very easily but it's not so great for file uploads because there's like caps on how much you can upload it's not so great for dealing with streams and that kind of thing when you're working with serverless so the the best way that i found from from reading is to use like a dedicated server um to to handle file uploads and that's what i'm doing here so i've got this upload endpoint uh it's accepts post requests and i'm using this library called busboy if you've been around node for a while you you might have used this or maybe you're at least familiar with it i've never used it until now i've never heard of it but it helps with streaming file uploads um so what happens is with busboy is instead of taking at your end point the whole file let's say it's like a gigabyte or two gigabytes or something that's going to chew up the memory of your server very quickly instead you can use busboy to take parts of the file at a time you can stream your uploads and that's what we're doing here so the this file here that comes through if you did like file.length you would see many instances of how many bytes are coming through at a time so it happens it happens in chunks and that's really useful for freeing up the memory on your server when once i get the file i upload it to s3 so taking in the file as parameters here i can then send it over to s3 once that's done i can add a job to my queue to that i'm using bull for for this queue um for so bullets like a message queue for for node it's very good you can you can add jobs to it and then process them in the background so my logic here is basically once the file gets uploaded to s3 as far as the user is concerned that upload process is complete for them so they can get a message back uh and i'm doing that here response.json message done this says to the user okay the upload is complete but there's still a processing step that needs to take place and so what i'll have eventually is some kind of record in my database that says a file exists an upload has been started and it's in a state of either like uploading or processing or whatnot and that information will be sent back to the user to see where their file is at and i'm using super base as my database host and they've got a lot of great stuff for real time so i'll probably tap into the real time functionality that they've got to to just up update the client in real time but in any case as far as the user's concerned once it's at s3 their upload is safe there in s3 it's done from the upload is done from their perspective um they get a message back saying it's done but there is still the the upload to muse which will happen in the background which will be kind of the processing step for the user so muse upload queue is down here you can process whatever comes through in the queue as a job and that's what i'm doing here i'm taking the job that comes through i'm using the same kind of method that i did on the client where i've got a new form data i'm appending the data to it and then i am sending that over to muse via axios here and that will take some time to do once it's done you'll get uh uploading complete and the eventually the database state will be updated to say that uh everything is is done being processed there's a bit of processing time once the file gets to muse as well and i can i can ping for that and see if that's done before sending messages back to to the client saying everything is completely done the idea is that once the file has totally been completely uploaded and processed they'll then be able to interact with it and use it like add it to courses and lead magnets and stuff like that um and that's that's the the general idea i'll show you one example of a file that's a bit bigger so this video i did a while back as 88.7 megabytes the first one we did was 11.6 this one 88.7 so if i open this once i do it will start uploading and so this progress bar should move along there we go 4052 it's it's moving along things are uploading and again this happens because of that really easy callback that we get from axios so axios was very nice choice in this case to show the actual uploading progress that the file has taken on once we have uploaded it we should get it here cascading deletes is the upload that went through to s3 it should now be on s3 let's check over here we get cascading delete it is there and we should see eventually because we're running with a cue things are happening in the background again from the client's perspective the the job is the the upload job is done they've at this point gotten a message back saying complete so i can show like a check mark here eventually but in the background on my node server i've got the file processing and in fact that's the state that we're at here processing file it's showing the info and then dub done uploading comes through once it's done so in theory we are now over at muse we should be there and there we go so we get our file coming through here and uh you know so here it is we got you know the transcription done for us this uh you know comes through very very quickly awesome stuff from muse i'm really excited to use this so my logic here is basically like s3 is kind of long-term storage for the file um i i kind of need a place to store the file before i send it over to muse anyway like it could be you know file storage on my server um but because servers are ephemeral i don't know if that works the best um you know but but this also serves as long-term storage for the file if it's over here in s3 and so if anything ever happens to the files on muse i can go back over to my s3 and you know send them somewhere else or whatever whatever the case might be file lens then ends up on muse where all this processing is done and then i can have students consume these files uh through like the web interface that they will see that the author creates for them so that's the general overall kind of uh picture of where things are at with the upload process there's there's more logic that needs to go in there there are kind of air conditions that i've definitely got to handle there's more to do here but that's i think the the base of it is in now so hopefully this was helpful um you know if you if you like this kind of thing where i give an overview of the architecture let me know in the comments if you want to see more stuff like this and um you know i'll try to do a few more of these things you can check out courselift if you're interested in the product itself it's at courselift.com uh or check it out on twitter it's at courselift there i'll put some links to to this in the description also i will put i'll try to do like a small demo of this architecture put it together on on github and i'll put a link to that as well alright thanks for watching

Channel: Ryan Chenkie

Views: 18,834

Keywords:

Id: scUKcl36ZQs

Length: 10min 38sec (638 seconds)

Published: Tue Aug 10 2021