How to perform multipart upload to S3 Bucket using AWS CLI

hey guys uh welcome back my name is Avinash and in this video I'm going to talk about uh an option called S3 multi-part upload all right so uh you know so if you want to upload the object to S3 platform we have multiple options like uh you can simply drag and drop an object here or you can use this uploader and you can upload an object or if you are using a command prompt or with AWS CLI tools so you can simply use AWS S3 space CP command and you could give your source file and you can give like a your S3 bucket right so that is going to copy all the data to S3 platform right so along with these options we do have an option called multi-part upload so basically we can make a large file into small chunks small parts and we can upload each and every part and we can combine that as an individual file so uh like you know whenever you are using AWS S3 space CP command so it by defaultly uses multi-part upload option so uh if you want to read about this you can use this specific article right so the AWS recommended one this is a recommended method like uh we can use this AWS S3 spec CP command whatever large file you want to upload you can just give that as a source file and you can upload it to your desired S3 bucket after configuring AWS CLI tools all right so um you know so basically it defaultly have multipart upload but there is a problem whenever you are using this method so if internet breaks in between so you have taken like a 10 GB file and you started uploading by using the CP command but after some time around five minutes suddenly internet break and you need to re-initiate that if it happened multiple times like we will frustrate right so what we can do with these multi-part option multi-part upload option we can break that 10 GB file into multiple parts so you can take it as a small chunks like a 1GB 1GB 10 files or else you can take 500 MB 500 mb20 files so we can make it small parts okay for that we can use a command called split command if you are using Linux operating system so we can make it into small parts and those small parts we can upload via multi-pad upload so simultaneously you open multiple command prompts and multiple objects you upload right so that is one option and if any one is failed you just read right right so we have an option um uh called multi-part upload to perform right so for that first you need to have AWS CLI Tools in my machine I already have awcli tools AWS space iPhone version you can give this command and it should return with the AWS install CLI tools if you don't have the CLI tools the first navigate to AWS Dot amazon.com CLI and based on your operating system you can download the required software from here and you can just install in your operating system in your device so once this installation is completed you need to do the AWS configuration just run AWS space configure when you click on enter it will ask you to enter access key ID secret access key default region and default output format so you just create an IM user whatever access key ID you get whatever the secret access key you're going to get just get that information and configure that here all right so if uh I already have these things in my local machine if you want me to go through this pretty quickly I can do it I'm just navigating to IIM and in left pane we have users add user I'm going to call this user as S3 test user is a name I am giving and just click on next and I'm going to attach a policy directly so we can give S3 full access to this user or you can give administrator access policy to this user so it's better to go with list privileges mechanism but for this demonstration video I am giving administrator access now the user created so let's go to security credentials scroll down we have an option called access Keys click on create access key so then select command line interface then select I understand the above recommended and want to proceed to create an access key click on next then click on create access key all right you observe here we got an access key ID and secret access key now once your system is displaying this whenever you execute this AWS space iPhone iPhone version if your system is displaying this install CLI version so then you are good to configure this access keyword and secret access key just give a command data plus space configure then click on enter as I told you it's asking me to enter access key ID let's graph that access key ID paste out here then it is asking me to enter secret access key I'm going to copy that secret access key I'm going to paste it here then click on enter default region I am good with Mumbai region so AP South iPhone 1 so then default output format none all right so now we are done with this Eli configuration so please note one point don't share this access key ID and secret access key with anyone so it just like you are giving your username and password to someone and um they can configure the CLI and they can see all the resources they may launch a server on your AWS account or they might they can even view all the resource they can delete the resources right so that's the reason it's always recommended to go with least privileges mechanism right um so now this part is completed so my requirement is I have a file on my desktop uh there is a folder called key pairs under that multi-part so if You observe there I have a video with a name called video.mp4 so if you give LS iPhone LH and you see this this file size is 400 to lmb right so upload this special to my S3 bucket so there is an S3 bucket with the name called Avis pay I want to upload this specific object to my S3 bucket by using multi-part upload all right so for that as I told you first we need to use split command so basically the split command is used to make this large file into small parts Even If You observe I'm into the same folder You observe here this is what video dot mp4 file is and I'm going to use the split command iPhone B I'm going to make it bytes so you can give 100 MB so it will make five files 400 MB files and one to lmb file so if you want to make it like a 200 MB files like you can give 200 MB now what happened it will make three files two 200 MB files and one uh 12 Mb file so we can give that 200 MB so it depends like you know how good internet uh you have it depends on that we can make it a small chunks so um I'll do one thing I'll go with 150 MB so then this video dot MP4 click on enter now uh if you give LSP space iPhone LH you see here two 150 MB files created at one one to lmb file created so technically this 4 to lmb is now made as a three parts and if You observe the names x a a x a b and X AC right so you can even verify that from browser so x a x a b and X AC all right so we make this other three parts now as a first step we need to initiate this multi-part upload so if you want to initiate the multi-part upload the first uh we need to create the multi-party so there is a command called AWS S3 API so then the command is create iPhone multi-part iPhone upload so which file you want to do and to what bucket you want to do so give if an iPhone bucket my bucket name is Avis Bay and what is a key which file you want to my key is Radio dot MP4 all right so you can give this command so if you're giving any typos at all it will throw an error so when you execute this create multi-part upload when you execute this create multipart upload command it's going to return with an I job ID so you know I will do one thing whatever the desired commands I required I'm going to paste it in this document so let's observe this this is the first command to get the upload ID so copy this command I'm pasting it here now we just need to replace this with to what bucket you want to upload you need to give the bucket name and what is the file name so in my case the video name is it's just video.mp4 radio dot MP4 all right so when we execute this command what happened as I told you it's going to give a job ID let's execute this command all right you see this we got an upload ID so this is very very important you need to save this for future usage so let's copy this so you can remove that colons it's not really required so I just removed it so this is my upload ID this upload ID we are going to use in future commands so save it for future commands all right so let's um so now this initiated a multi-part upload so now whatever the individual files we have here xaax ab and X AC so these three we need to upload it now to S3 bucket and once all the files got uploaded we need to complete that multipart upload okay so for that we have commands here so take this command so you can exclude this md5 checksum right so I'm going to copy this right let's replace it so this is what so bucket name is uh of this way and key is video dot MP4 and part number I'm going to upload it as a one and iPhone iPhone body here you need to give whatever the file names like you got it here like xaax Bay a b and X AC right so that you need to give so I'm going to give x a a as a file name and upload ID whatever the one you got sometime back right you just replace it here it's just an example ID you replace it so this is for first file you I'll do one thing I will create remaining two walls so this is going to be part number two x a b so then take the part number three and it's going to be X AC so we have three files x a b x a b and X AC so I created this command for all the three so you know if the this part size is huge we can simultaneously execute these commands but it's very like you know 150 MB right so I'm going to execute it one by one let me clear the screen here all right I am going to paste this click on enter it uh it returns with the e-tag so we need to grab that e tag also okay so the command is executing so it won't take um much time it's just 150 MB right and whatever the upload we are performing now we cannot see those content until unless you run that complete multi-part option you just observe here even I enable versioning disable versioning so I don't have any data within this bucket all right so the first part is now uploaded you see this part one I've uploaded this e-tag is important for us so let me do one thing let me copy this e-tag let me write it just below to that first part and similar way we need to execute this second command and we can take a new terminal and we can even take that third part and we can execute that here okay I'm in different path now so I need to go to desktop keypads multi-part right so basically it's saying you are pointing to a file called xse but where we are working in that location that is not available right so yeah now I'm executing the third part one also in different terminal right we got um second part e tag let's write down that second part e tag and once that is completed let's take that third file you see part number three e-tag so we need this e tags and we need to prepare a simple Json file by using this e-tags all right so now we got first a tag second data again 30 tags what we can do with the zetax so we need to prepare a simple Json file to combine these three parts so how AWS know which one is the part one which one is part two which one is part three right so AWS Knows by the document we are going to prepare now so if you observe here uh this is the document format Json format all right so but before that if you observe if you want to verify whatever the commands you have executed like uh whether all those uh files uploaded to S3 platform or not so you can use an option called list parts so let's give a try so let's uh take the previous command okay right instead of upload part just give list part okay so it should show all the individual parts we have uploaded let me clear the screen now I'm going to execute that command click on enter it's list Parts I have given a typo I have just given list part its list parts all right just execute that command click on enter oh this body not really required part not really required okay um let me cancel it all right I okay just we just need key and upload ID execute it now so you see here it's saying there is part number one part number two and part number three so with this like a working as a S3 test user that information we can see here all right so the list part is now showing all the available three parts okay so now let's execute uh a command to complete this multi-part operation so for that as I told you there is uh like you know this is a Json syntax we need to add that e-tax so let me write down that here so first part this is like this is syntax I'm going to do this all right this is a part one and I'm going to create part two we have total three parts right okay we are good with this this is part one this is part two and this is going to be part three okay part one e tag just copy this and replace this part two e-tag copy it and replace it here and similar Way Part 3 also just copy and paste it here so you need to create this as a dedicated file so let me open a new file I'm going to paste it here I'm going to save this to the same same location I have keypads multi-part and I'm going to call this as foreign script so we need to pass this file name while we are executing next command so let me copy that all right so the file now created so this is the file name we just need to give file colon slash slash while we are passing this okay so now the last command is to complete that multi-part just copy this command and replace it with required information so we are completing the multi-part upload why because we have uploaded all small chunks right so I'm going to replace this file with my own file and bucket name is obvious way and key is nothing but what is a file name so my file name is video dot MP4 and also the upload ID whatever the upload ID we got when we created this multi-part you just need to replace it here okay so now execute this command so make sure where from where you are executing this file is available in the same location otherwise it will throw an error all right in my case I have that file in this location let's paste that command and enter so now your output should be this so it should show the bucket name it should show the uh object name and it should show the entire location of that bucket so now if you go to your S3 platform if you give a refresh you see this video dot MP4 is now uploaded and the size is same as our original one okay so with this way we can like you know if it just a 400 MB file I just want to make this video a bit faster that's the reason I took a 400 MB file but if you are dealing with large files we can use this option and one more point so sometimes what happened you have initiated multi-part and some multi-part upload fails and some chunks failed if you want to clear such type of mult unsuccessful multi-part upload files you do have an option to clean up that so for that go to the S3 bucket management create lifecycle role give a name and just select apply to all data and you see this delete expired object delete markers or incomplete multipart uploads so when you select this option like you know whatever the day is delayed incomplete multipart uploads so that whatever multipart upload operation you have initiated so that should complete within one day or that should complete within two days or whatever days you mentioned here if your multi-part operation is not complete within this given days it's going to delete all the incomplete Parts like the chunks we have uploaded right so those all chunks are going to be deleted automatically so for that we can go to lifecycle management role and select the last option and delete incomplete multipart uploads all right so that is all about multi-part upload I hope you uh I hope you find this video informative so thanks for watching this video please subscribe thank you guys

Channel: AWS with Avinash Reddy

Views: 7,072

Keywords: aws with avinash, avinash reddy aws, naresh technologies, aws naresh it, naresh it ameerpet

Id: ZJaNTLKz334

Length: 23min 6sec (1386 seconds)

Published: Wed Feb 22 2023