How to upload files to AWS S3 with FastAPI ?

hello everyone this is Satish I am a junior developer I've recently struggled a lot to figure this out I took a lot of preparation and learnings to create an endpoint in fast API which takes a file as an upload file and then a pushes it to AWS S3 bucket so uh I just wanna I just want to share that with you so that it would help you in the future so uh here's the flow uh we are gonna have a front-end client which is gonna call the upload endpoint of our fast API backend and our first API is just gonna do some pre-processing like uh in our case we are just gonna check the file size and uh the actual type of the file contents and then if the contents and the size are ideal and according to your requirements then we'll just push it to our AWS S3 bucket so that's the flow and if you look at the application into the programming part we just have a simple fast API app which has a root endpoint with a nice Hello message and then a second endpoint which is just the upload endpoint which should be implemented obviously which we are going to do now let me just run the uh application reality so you can see it is running now if we just hit the home endpoint we get a nice Hello message okay so now we have the application running let's look at what we are doing in the upload endpoint so this is the parameter the file parameter which is automatically gonna take the file contents which is gonna provide an upload button in the Swagger URL also which is the nice thing in fast API we get so the uploaded file contents will be in bytes format and will be put in the file object so this is a an edge case that we are handling here just checking if the file contents are not none and if the file contents are none we are just responding with an HTTP exception and the status code is 400 and no file formed okay so now if the file is not none then we are gonna start processing it all right so first we are gonna read the contents contents of the file like this it's an asynchronous method so we need to await that file.3 and this is gonna have all the uh contents in white format so once we have that I'm just gonna paste a few constants that I have which we can use for our convenience because we are gonna checking the size it's good to have these variables so a KB is one one zero two four bytes and similarly NMB is one zero two for KB so we just it's nice to have the constants like that already so now uh the size let's get the file size it is nothing but the just the length of the contents right content contents is just a byte stream and total bytes is just our file size so once we have the file size we can go ahead and uh put a check there I'm just gonna paste that to not waste time so if the file size let me just change that to size real quick okay if the size is less than an MB and greater than zero of course we don't accept files which are of zero size and then response respond with a status code 400 and the reason which we are rejecting the file so once the file size is uh in the desired range that we are supporting we can go ahead and extract the file type so the file type we might think that it's okay to extract the file type from the uploaded file name but that's not a good idea and that's one of the vulnerabilities in file uploads so people what people do is they might tend to inject some scripts into your backend like let's say a HTML file or maybe some scripts they try to execute them by renaming them or changing the file extension from dot HTML to maybe dot PNG or maybe Dot php2 some other file types so that's gonna create a vulnerability and execute unwanted scripts in your uh backend which we don't want so we don't rely on the file types that the user provided and we extract our own file type by using a library called Magic next import that you can install that with the PIP install python hyphen Magic so once we have that we can get the file type from buffer because we have a byte stream we can use from buffer method of magic and we can provide the buffer and provide the contents of our bytes and also we can say that mime is equal to true which is basically uh telling the function to return the file type in a specific format so it just Returns the file type in this format or else it's gonna return in a human readable format which we don't need in this case we already have a nice dictionary of supported file types we are supporting PNG jpegs and PDFs so to get the output in this format we just need to give the Flag mine is equal to true so now we have the file type and we are going to add another check here from the file type which is this so if the file type is not in the supported file types dictionary I'm just again raising an exception and similarly which I did here so with a reason why we rejected the request great now we have done two checks one is the file size and the file type we did not rely on the file name we just extracted it ourselves with the help of magic and now we are good to upload our file to our S3 I'm just going to create a function called S3 upload which is gonna take contents it's going to be our contents and a key which is gonna be uh a string now what is gonna be the key so the key you can use it as your file name which you can directly take the user provided file name but as I said it's not a good idea to have a user provided file names you can store it separately somewhere in your database and put the actual file name as a unique ID and then link them both while providing a download and point to the user so if the user want to download the file back again they can have the same file name that they have provided but in our case I'm just gonna use a unique ID so the file name is gonna have a unique ID as the file name and the extension uh the appropriate extension which we know from the file type so the extension is gonna be supported file types of file depth so this is basically giving these strings like Dot jpg.jpg.png.pdf okay so the file name is gonna be a unique ID which I'm gonna generate with this function which is also python provided from uuid library so I'm gonna just import that from uvid import which is nicely providing you IDs for us and now let's uh write this function okay I have already have it written and I'll just paste that to save some time okay so as you can see this is my AWS bucket and I have that in this variable constant and as you can see there there are no objects in in the bucket right now and we are gonna use a library called in Moto 3 which is provided by AWS themselves for uh handling these kind of things with AWS so from code 3 we can extract the resource S3 and from S3 we need the bucket with our bucket name so this is gonna get the instance of our bucket so that we can manipulate our bucket up put files to it and get files from that and I'm just gonna log things out using logaru from logaru import logger so this function is just taking our bytes and our key I'm just gonna rename that to key and then put the object into the bucket so we have the bucket already it's just gonna put the object the file object into the bucket with the key has the file name which we provided a unique ID and the extension okay so we are good to go let's look for any errors yeah we don't have any errors uh we see that the application is running and let me just go ahead and execute it's gonna refresh try it out and then select a file I'm selecting this file which is of 61 KB which is less than 1 MB and greater than 0 KB I'm selecting that as you can see it is selected and execute okay we can see that okay the response is 200 and you can also see that the output message which we said here uploading the file name as you can see the file name is PNG and a unique uniquely generated ID uuid okay let's check if the file is actually uploaded to S3 or not okay as you can see that we have uploaded a file here okay let's uh just providing the download link here as we have downloaded that let me open that which we are open yes as you can see let me have uploaded successfully to S3 using first API that's all for today guys hope you liked it bye

Channel: Jr. Developer

Views: 6,444

Keywords: fastapi, aws s3, file uploads, python

Id: 727l8Asu8P0

Length: 13min 5sec (785 seconds)

Published: Thu Dec 01 2022