Chunked File Upload with React and Express.js

hello everyone today i'm going to show you how to create this chunked upload with progress bar using express.js and react and as always we start with an empty directory and so now we can create a react part so let's create the npx create react app and let's call it client okay now so as you can see we have it installed in client directory as a subdirectory of our main directory so now we can go inside client and we can start our react app okay we have our react app now let's go to client source and app.js and maybe let's remove some stuff we will not need so let's just create an empty div here like this yes and we can also remove this logo yes and let's add test here so we know it works okay so the first thing we want to do is to create a drop zone for our files so let's try with div job drop zone yes and now let's go to app css let's remove all those because all the load stacks because we will not need them now let's go back to our js and let's add the take some here drop your files here yes and now in our app.js let's add some styles okay so let's let's start from the background i like some dark colors so let's do background color maybe one one three yes and the color of the text needs to be brighter so let's do a ddd like david and let's add also some padding i don't know maybe 20 pixels all right now with the drop zone let's add this dotted border to pixels dashed and let's do rgba and let's make it 20 transparent all right so now let's do some border radius of let's say 10 pixels and padding from the top and the bottom of 50 pixels and from the size it will be zero now let's center our text text align center oops it should be a line and let's do a text transform uppercase and now let's make it more transparent i mean the text let's try with uh 60 all right now let's go back to our app.js and in here we need some we need to add some events to our drop zone all right so let's do on drag over first on drag over we will get an event and with this event we want to tell the drop zone that we have a file over the dropped zone so we will do something like set the current state of the drop zone to active so to do this i will create an empty function for now and at the top our import use state from react so we can use state here let's call it drop zone active and set drop zone active equals use state and the default value will be false oops set job zone active and when we have a file over we want to do set drop zone active to true and we also want to do prevent default behavior next on drag leave we want to do a similar thing so set drop zone active to false and we'll also prevent the default behavior here for on drag live event and the last event we need to handle is on drop so when someone drops file or files into our drop zone and here we need a specific function for this so let's create it here let's call it handle drop and it's a function and we'll pass an event here so now we can use it here handle drop with the event okay first thing is that we want to prevent default behavior of this event then let's use this drop zone active so where we can change the border colors when there is file over it so let's see we have class name drop zone but let's also add the if drop zone is active then we will add active otherwise we'll add nothing and in our app.js let's try jobzone active and let's do a border color of white now let's test it and as you can see it's changing the color when the file is over right so let's now fix our handle draw function so inside this event we have actually array of files that we drop on this area so we need to save them somewhere for now let's use state so let's do const files and set files equals use state and default value will be an empty array so here we want to do set files and we have files inside event data transfer files but we don't want to replace existing events we don't want to replace the existing files if we drop new files instead we want to add those new files to the existing files so when we drop new files we don't want to set those as files replacing the old ones we just want to one we just want to add them as new files so to do this we need to do set files with existing files and the new files so to do this i will create an array like this and i will distract this array like this and i will also add existing files here like this maybe i will replace the order of them like this okay so now when we drop files we want to do something with them so let's use effect and use effect here if files length changes then we want to upload those files or at least start uploading and this should be an array okay so if we have some files let's check if file's length is bigger than zero and we want also to check if we are not uploading anything right now so how to know if we are uploading something let's put it as a state here so const let's call it maybe current file index and set current file index will be use state and the default value will be null so this current file index will have an index of the file in this files array of the file that is currently uploading so we will check if we don't have anything here so current file index is null so we know nothing is uploading right now and if so we want to start uploading the first file so let's do a set current file index of the first file but it will not be always first filed we it will be first file only on the beginning and later when we drop profiles it should be the next file so the next file is 0 for from the beginning but later we want to know how many files we have already uploaded so to do this let's add the last uploaded file index and default value will be new and in here let's check if last uploaded file index is no then it will be zero because we don't we haven't upload any files but if we did then then the current file index will be last uploaded file index plus one so the next file will be uploaded let me only format this yes like this okay so now we are updating this current file index when we get new files here so let's also monitor the change of this current file index using use effect so let's use effect empty function for now and in this array we will do current file index so in this function we want to start uploading this file of this index so to do this we want to tell the app to start uploading and we start uploading by uploading the first chunk or the first part of the file so let's maybe use also state for this i will call it current change index and the default value will be also null so when current file index changes when there is a new file to upload let's update the current check index so the part of the file that should be uploaded will be zero so let's do set current chain index to zero but before let's also check if the current file index is not empty so current file index is different than null then we start uploading the first chunk of the file okay so now we want to know when this current chunk index changes so let's use another effect and the current chunk index will monitor here and in here we'll add an empty function for now so in here we want to just upload this current chunk index so let's just check if it's not null so current change index is different than no then we want to upload this chunk of the file so to do this i will create a function for this function read and upload current tank and i will use it here read and upload current chunk okay so in this function we want to take file from files with this current file index and read this current chunk index and we want to upload this chunk of the file so to do this we will use reader api functionality so let's do const reader equals new file reader now this now we need to tell the reader what part of the file we want to read so let's do first const file the file is files of this index so current file index now just in case let's do if there's no file let's just stop here and otherwise we want to know what part of the file we want to read so let's do const from so at the beginning it will be zero but for later chunks we want to read a specific part so let's first define how big the tanks will be so let's do const maybe here chang size and let's try with maybe 10 megabytes so 10 10 kilobytes so 10 times two like this so we have 10 kilobytes here the chunk size will be 10 kilobytes all right so back to our rain and upload time so the from will be current time index times change size so for the first chunk so the chunk of the index of zero it will be zero times um 10 kilobytes so it will be zero but for for example the first one we will start from 10 000 kilobyte of the file right so now const 2 will be the ending part of the of this chunk so it will be from plus the chunk size like this so we will always read from plus the chunk size okay now let's take binary data that we want to read so const blob equals file and we need to slice this file from two all right now we need to tell the reader what to do after it's done so let's do reader on load and it will be an event here and later with the part of the file that we have read let's upload it so to upload a specific binary part of a file i will create another function and it's going to upload chunk and we will pass this event here and we'll use it here upload change like this and at the end we just need to tell the reader to start training read as that the url will be fine and here will be the blog okay so here let's maybe rename this to reader event here we want to know what file we are reading so let's do cons file equals files and current file index like this and we want to get the binary data so let's do const data equals reader event target result okay now i want to send it with post request to our api and to do this i will use axios so i will open new tab in my terminal go to client directory and i will add yarn add axios like this okay and i will also import it at the top here import axios from axios and back to upload chunk function now we want to do axios post and we want to post our data to our api endpoint so let's say it will be http localhost 4000 slash upload and in the data we want to put this data here so it's the same as here and we also want to send some extra params like file name and the current progress of the upload what is the current chunk of the file what part of the file is it so i will do const params here that will be an empty object for now and i want to add it as a params here so let's do question mark plus and let's do params to string and instead of using an empty object i will use new url search params like this so now we can do like paragrams set name will be file dot name and also paragrams set size will be file size let's also send the current chunk size maybe paragrams set current i mean current chunk index current chang index like this so we know what part of the file we are actually sending is it the beginning of the fire or is it the end or is it somewhere in the middle oops it should be current chunk index like this and we also want to tell how many parts will there be so is it the first and the only part of the file will be there any more parts of the file so let's do programs set total chunks and this will be file size by size divided by the chunk size and this can be not an even number so for example 2.5 so to fix it we'll do math sale so if it's 250 megabytes it will be actually three parts three chunks of the file if we have 100 megabyte chunk size so that's why we are using math sale to round to the top so it will be like three parts instead of 2.5 okay now let's also tell our api that this is not a regular api json request but actually a file that we are sending so let's create some extra headers content type will be application octet stream yes it's fine and we'll add those headers here and to format it better i will do url here like this okay so after sending the post we'll have a response and with this response we will do let's fix our backend first so here we have our client now we need to add the backend part so let's create our api so let's go back one directory and and let's create the api directory now we have it here and let's go inside our api directory and let's yarn add express and let's create a new file here called server.js and here we will put our express app so the first thing i want to change that type always to module so i can do esx imports and here let's do import express from express like this and now let's define our app so const app equals express now we can define our upload endpoint so if someone posts at upload we will we want to do something with the request and the response and we need to start listening 4000 port okay so when someone posts at upload we want to take the chunk that the that someone is uploading and save it somewhere inside our files somewhere between our files here so i will create a new directory here for our uploads like this so we can upload so we can upload our files here first let's get all the params so from the request we are sending params like name size current [Music] tank index and total chunks so from the name we can take the extension of the file to know if it's jpeg or something else and because we will need it later so let's do const extension equals name split let's split by dot and let's pop the last part so if it's a test.jpg we will just take the last part okay so now to take the row body of our post request we will need something called body parser so let's yarn at body parser to our api and now let's import body parser from body parser yes and in our app let's just use both the parser row row the type is application slash octet stream and now we need also to define limit let's say it will be 100 megabytes oops it should be like this okay so now let's get the binary data that we are sending and let's see what do we have here there inside let's do console log and request body and just let's send raise bones send empty string okay let's start notebon with our server.js oops this is already news let's try with four thousand one now it's better and now just let's change this four thousand here to four thousand one and let's see what happens if we try to upload something nothing happens let's see what do we have here okay so we have your problem so we have a course problem because we need to define some course policy and because we are sending post requests between different domains we are at localhost 3000 main and we want to send the localhost 4001. so to fix it let's go to our api oops api and let's yarn the course and here in our server.js let's just import course from course and let's just use it here of course the origin of their requests will be http localhost 3000 now let's try again okay so now it's frame five 500 error let's see cannot read property split of undefined yes this is because we it's not the request where we have all the params it's in request query like this now let's try again i will refresh let's upload yeah so we have some bytes here that's maybe called to string and let's try again okay here are the bytes that we are sending but on the beginning we have this meta information about the type of the file we are sending and we want to remove it from the binary data so as you can see it's divided from the rest with the with a comma so to get just the rest of it this i will do split by the comma and i will take the second part so index one so i will take everything starting from the comma so now instead of console logging let's remove this and let's assign it to a constant called data like this now to save this we want to change our data to a buffer because otherwise it will not be possible so let's do a const buffer equals new buffer and let's put our data like this okay so now we can put our binary data to a file in side uploads directory so to do this we need to import fs from fs so it's file so now we can do some stuff on file system so the easiest way will be fs that append file sync let's do synchronously and then we could do here uploads slash and then the file and the data here not the data the buffer like this and this will work the problem is that when we upload a file with the same file name it will overwrite the previous file it will put and the new file at the same place as the old file so we want to randomize the file names that's why we need this extension let's start with a temporary file name temp file name and this will be let's randomize it so let's add maybe yarn add md5 and now also let's import md5 from md5 like this so the temponary file name will be md5 off maybe the file name plus the dot plus the extension let's maybe also add here request ip and here instead of putting it as user file name let's use our temp file name okay now we want to use those information about chunks so i want to also know if it's the first chunk or the last change of the file so first chunk will be if current chunk index is and zero just use parse end of current tank index is zero and const last tank is when current chunk index is same as total chunks minus one so if we have five chunks the last chunk index will be four that's why we'll have four here five here minus one will be four so uh last chunk will be true okay so if it's the first chunk first chunk then let's clear some place so fs and link sync let's clear if there is any file it displays uploads slash and tmp file name like this then we put this binary data inside this file then if it's the last chunk we want to rename this file and put it maybe somewhere else i know let's do that temporary file name will start always with team p underscore like this and if it's the last chunk we will do a const final file name and it will be randomized really randomized so date now to use some timestamp here and let's add also let's maybe do sub string of this to make it shorter from zero to maybe six let's add dot and extension at the end so it will be that jpeg or that's zip and we want to rename from tmp underscore something to our final file name without tmp underscore so let's do fs rename sync from uploads tmp and tmp file name to uploads final file name like this and i see that we don't need this size parameter here so i can remove it okay now let's go back to our app.js so here inside our react app we are reading and uploading the current tank so here we are uploading the chang but after we uploaded the first chunk we are not doing anything so after uploading the first part of the file we want to start uploading the next part okay so first we want to know what is the next part that we want to upload so let's first check if it's the last chunk the last part of the file is last tank so we want to know if the current chunk index is the last tank index so equals the number of chunks minus one and the number of chunks is actually math sale of file size divided by chunk size and the file size we don't have it here so let's do const file size equals files current file index size oops const okay if it's the last file chunk so is last chunk then we want to know what is defined my final file name so let's maybe get the file const file equals files current file index and if it's the last part of the file we want to know the final file name and but we are not sending it as a response so let's go back to our server js and let's see we are sending only okay so here if it's the last chunk let's response send maybe let's use json and he'll final filename and here just jason of okay okay response send okay so now the final four name will be from our response data final file name like this so if we have upload that the last chunk of the file we want to upload the next file so let's first notify the state the last file index that we have uploaded so set last uploaded file index will be current file index and let's also set current chunk index to no so this will be cleared too and if it's not the last part we want to start uploading the next part so to do this we need to update a current chunk index so let's do a set current chunk index and here we will the current chunk index plus one because we know that it's not the last one okay so here we are setting the last uploaded file index but we are not checking the changes of this state anywhere so let's maybe add it here so use state and empty function here and we will check last uploaded file index so this will run when a file finish uploading when the file finishes uploading so when the file finishes uploading we want to start uploading [Music] the next file if there is any so now first let's check if the last uploaded file index is not null because if it's no we don't want to do anything so let's do return nothing but otherwise we want to check if it's the last file so is if the uploaded file is the last file so is last file and to know if it's the last file we can check if the last upload filed index is the current file index or better files length so and minus one yes so if the last uploaded file index so for example if we upload three files this will be free and the last uploaded index will be two because there are three files so the last index of the error will be 2 and will be 2 here will be also 2 because 3 minus 1 is 2 then we will know this is the last file so now we can define next file index and if it's the last file then it will be no because we don't have anything any other files to upload but otherwise it will be current file index plus one so if it's not the last file then we can upload the next one but now let's use this const next file index and let's put it as set current file index here next file index like this okay let's try let's maybe start with a one file and jpeg and we have another let's see no such file or directory when we are unlinking okay so in our server desk we are unlinking so removing the file if it's the first part so if it's the first part of the file we want to remove this file if there is something else taking this place of this file so we want to clear the space before starting uploading the first part of the file so but when there isn't any file there we don't want to do this so uh if it's the first chunk and we need to check if there is something there so file it's exist sync and we want to know if there is something so let's copy this part and put it here so if it's the first chunk and we have something at this place we want to remove it before we start uploading now let's test again okay something happened let's see yes we are uploading some stuff here but we are not displaying any progress or any files here so to do this let's go back to our app.js and as you can see we have a drop zone here but under the drop zone we don't have anything else let me just put it maybe like here and here let's put the list of the files that we have uploaded so let's do div dot files okay so here let's list our files so files map and here we have file and let's create a function here let's do return and leave file like this and maybe let's put name of the file somewhere here so let's do this name and here file that name okay now instead of using dave's let's use a link here so let's do target blank and let's use graph file final file name and on the beginning we need to add the http local host 4000 slash uploads slash let's test it okay it kind of works now we also want to see the progress of the upload so let's do let progress equals zero from the beginning then we also want to know the file index here so here if the file is uploaded so file final file name is inside the file then we know that the progress is 100 but otherwise we want to check what is the progress of the upload so first let's check if we are currently uploading this file const uploading will be file index equals current file index and if we are uploading then let's say that the progress is 50 percent otherwise it's zero because we haven't started uploading it and let's just put it somewhere here to see how it looks like so this one is 100 let's try again let's see if it gets 5 50 if it's uploading yes it gets 50 right so now we want to change this 50 to the actual progress so we know what is the current chunk index and we know how many chunks do we have in this file so let's do const chunks this will be file dot size divided by chunk size and let's also do math sale here and here we can use chunks let's test again yeah the one thing i forgot is to do times 100 and the math round let's test again now it's much better okay now let's only put some styles to our list of the files so let's do that file that detects the correlation none so we will not have this underline and the color of i don't know white ish that's 80 of white display block let's do some background color of let's say 41.41 5d like this and some border radius often pixels margin top of 20 let's also add some padding maybe 10 pixels here because let's use position red relative and overflow will be hidden okay now let's do file progress and say that the background will be the background of the progress bar will be 80 80 120 oops i forgot the comma here and position will be absolute inset will be zero so now as you can see we have this hundred percent here yeah let's add this percentage sign here and we want also to edit the width of this div and while it's uploading so let's do style with progress and let's add percentage here so let's see let's test again yeah now it's growing here okay now i need to add some padding here from the top and the bottom so let's do maybe padding top of 10 pixels like this and the text line center and also the default with will be zero percent or just zero and let's test it again yeah it works but when it's done i want to remove this hundred percent so i will only have this file name so i will maybe add another class let's go back to fjs let's see here we have our progress and i will add the new class oops i will put it like this and the new class will be if the progress is 100 then i will add done otherwise i will not add anything so if it's 100 then it will be also done so if i go back to app css and i add the file progress done i can do top 100 100 pixels yeah so it's removed from the view yeah so it got moved to the bottom so it's not visible because we have this overflow hidden okay now let's try again yeah and if we click it oh we got an error and this error is because in our server.js we are not serving those uploads files and to do this we just need to do app use and here let's define direct euro so everything that starts with uploads we will do express static uploads now let's refresh and it doesn't work okay this is because we are using wrong port here it should be four thousand one so let's fix it in our urls here let's refresh upload click yeah it kinda works okay one small problem i forgot that in buffer we need to define that this is base64 data so now if i upload again we have the full picture now let's only try to upload several pictures or several files and we have a small issue that after the first file is uploaded the next file is not uploading okay i see whatever it shouldn't be your state it should be use effect okay that's why it was buggy let's try again yeah now it works so that's all for today don't forget to click the like button and subscribe if you want to see more videos like this one

Channel: Coding With Dawid

Views: 1,971

Keywords: chunked file upload, chunked upload, express file upload, express js chunked upload, express js file upload, mern chunked upload, react js chunked upload, react js upload with progress bar, mern file upload, express js react file upload, express js react js file upload, express js chunk file upload, mern chunk file upload, react js chunk file upload, mern big file upload, mern file upload progress bar, express js upload progress bar

Id: dbYBVbrDnwg

Length: 58min 0sec (3480 seconds)

Published: Wed Jul 21 2021