Uploading Files to AWS S3 using Python and Boto3

hello everyone it's Steve from Financial Python and I wanted to put together this quick video on how to upload files to S3 using python so this isn't necessarily Finance related but it's something that I do a lot uh basically in all of my scripts what I do is I upload my log files into S3 and it allows me to you know organize them and view them at later dates if I need to uh you know kind of see if there's any errors happening or anything like that I did make a previous video on logging if you want to go check that out but this video is just going to be about how to take a file and to upload that into S3 so um yeah so let's get started on the AWS side so the first thing we need to do before we write our script is do some configuration in AWS so I'm going to go to S3 where I'm going to create a new bucket so let me create a bucket here and I'll just call it FP test bucket um and I don't believe I need to make any other changes I am going to note my region here we see us East one but we'll get to that a little later what we're going to do with that so really I don't have to change any other settings I'm just going to do create bucket and oh contains invalid characters so let me fix that if oh name already exists there we go it needs to be unique so we have SP test bucket1 and let me create this okay so now that I have my bucket created um let me just duplicate this and I'm going to create a different uh AWS tab I just want to keep that open but I'm gonna go to a new service I'm going to go to IAM which is identity access management basically controls um you know permissions and users for actually accessing accessing AWS services so in here what we're going to do is we need to create a policy that says um you know a policy that grants access to that bucket and then what we're going to do is take that policy and we're going to give that to a new user that we create so let's go over to policies and I'm going to do create policy okay and what I like to do you see like if you can do some point and click configuration to set up your policy or we can switch over to Json and that's what I like to do I like to switch to Json and um and basically configure my my bucket permit or my policy permissions here so the first thing you'll notice is in Sid so you see it gives you a you know it kind of gives you a template for a policy here that we're able to edit and the first thing is Sid which is really just a unique identifier we don't actually even need it but I'll just populate it you know we'll say SP test bucket access policy but again not not even any something that we actually need um okay and here it's going to say what uh actions do we want to allow for this policy so you see like the policy the effect we're giving is allow and what actions are we allowing so I don't need an array I can just add in a string and it's going to be S3 with a star saying give me access to all S3 functionality and the resource is saying where do we want to give that access so we want to give it to that bucket so we want to in order to specify the resource again we don't need an array we can just do one string since only one resource and the resource will be Arn so Arn stands for Amazon resource name AWS S3 and this is just a standard AWS syntax to specify a resource and a policy three colons as you can see here and then the bucket name so SP we need to SP test bucket one and then if I give a slash and a star that's saying you know give access to everything inside of that bucket okay so that should be good so I'm going to click next and then I can name this policy so you see uh you know kind of the permissions defined in this policy is S3 uh read read list permissions um basically can do everything and the bucket name is saying you know string like FB test one and um yeah we know that's the bucket we named it so I can give it a policy name so we'll say FB test bucket policy call whatever we want there add description gives access to FP test bucket S3 bucket and um so we'll say create policy and we have this policy created that's great so that policy again what that is is that policy is going to give access to that S3 bucket but who are we going to give that access to that's the next step so let's go over to users and let's create a new user and the username we'll call it SP test bucket user again we call whatever we want we don't want to give access to the Management console we're not doing that so I'm just going to click next okay and as you can see the next step is going to be setting the permissions and it gives some different options about how we can set those permissions one of them being to attach the policies directly so I'm going to do that because we just created our policy and we want to attach that policy to our user so I'm going to search for this policy this SP test bucket policy we just created that's great I'm going to click next do we want to give any tags uh no we do not that's not necessary at this time so we can just do a little bit of a review and click create user okay great so now let me go to this user fptest bucket user okay and so we have our user created and we see our permissions for that user is this policy that we generated which is great and as you can see there's this other other tab here called security credentials and I'm going to navigate to that and you see here with that we have this um create access key so I'm gonna actually first before I do that I'm gonna go over to my code and we'll start making our script before we do this because we're going to need a place to put these after we create it so let's do that for a second so I have this file uh this folder that I created S3 upload and um Let me let me play around with that so let me create a new file in here I'm going to just call it main.py that's what I like to call all of my files let me I'll just uh do the standard python syntax of um of calling a method on on load on script run so when the script runs and the name is Main we're going to call a main method that we create now so let me create that main method and just to start I'll just say I'm in the main method um so right now I'm just doing standard python housekeeping but uh just because I wanted a place to load in these details after I generate them so let's actually do that real quick let's go to create access key when we're in this user and it's going to ask you know what our use case is and you know basically our use case is just local code at this time we're going to use this ad we're going to use this key in local code to access our environment I'm going to say I understand this recommendation and I'm going to click next I don't need any tags here so I'm just going to click next and as you can see what was generated was an access key and a secret key and I'm going to need both of these in order to programmatically access this bucket essentially you know with these keys I'll be able to log in as this user from my script so let me copy that and I'm just going to create a variable here we'll call it AWS access key typically I would have these in some sort of config file or something like that but for the time being I'm just going to put it right here give you a secret key okay so we see the secret let me copy that back to code okay great so we have that and again I just created that just so I could um uh I created this file and just started you know just wanted to start putting in my my details so the other details I'm going to need for this is the S3 bucket name which we generated so we called it FP test bucket one and I'm also going to need my region so I'm going to say AWS region and if I go back to AWS you can see the region is in your url so mine is us-east dash one yours might be different depending on where you're located uh but you are going to need you know the region that that you're located which is going to end up being the same region that your bucket is in so you see you know my buckets are all in this USC's one so that's where my my new bucket is as well so basically yeah that region is going to be wherever your bucket is located okay so now we have our AWS details all our AWS config done how are we going to actually upload a file so if I go to this documentation for voto3 so basically this package is an AWS python package if I just go to the main page here you can see this package allows you to create configure and manage AWS services such as you know these including S3 basically all of the S3 Services you can access using photo 3 and this one piece of documentation which I'll link in this video is an example of how to use how to use Moto 3 to do an S3 upload so we see you know we have the S3 upload here it does give some you know details in terms of like error handling we're not going to do that at this time and we do need some more details in this configuration which I'll go through as well but uh so let's use this example in our code and I'll talk through everything that we're doing so the first thing is importing the necessary package so you see there's importing photo 3 and if you don't have this already you may need to install you may need to install that by just doing pip install that's something I already have so I don't have to do that but if I wanted to again it would just be on the command line running you know hit three install photo three again I'm not going to do that right now because I already have it but that's how you would install that okay and with that the next thing we would do is is create our S3 client variable and set it equal to boto3 dot client and again there in this example they're only inputting S3 but we need to provide a few more details here so let's walk through what they are so I'm going to specify a service name and that is going to be S3 and then it's going to ask for a region name and we have our AWS region variable again I just have all my variables here typically and have that in some sort of uh config but I'm not going to do that for this video and then I'm gonna you do my um my programmatic access using the keys that we had generated so we have this AWS access key which we have a variable for and AWS secret access key which we also have a variable for okay and then from here we can actually do the same uh logic that they have in their try here like I'm just going to copy and paste that and explain what's happening here so um so again so what this is doing is taking this client that we generated and running the upload file method so that's something that's built in when we generate this photo 3 client with a service of S3 it comes with this method upload file and as you can see that takes a few different parameters so the bucket we already have right so we AWS bucket and then there's two there's two other variables that are included I'm just going to expand this so there's file name and object name and the difference between those are so the the file name is saying what local file do we want to input so let's actually create one real quick so I'm just going to create a new file I'm going to call it test file.txt I'm going to say uh here is my file I want to upload okay let's save that and if I go back to main I'm just going to create another variable called local file and I'll call it test file Dot txt okay and the object name is saying what do we want this to be called when it's uploaded into AWS so you know even because it's called test file here it doesn't necessarily need to be called test file when we upload it but we can just to keep it consistent and I usually do that again just for consistencies but you know I just as an example I'll create another file oh I'm sorry I'll create another variable and I'll call it name for S3 and we'll just keep the name for the same but I just wanted to create two variables just so you have an understanding about you know what those actually do and um from there we can just print our response which um kind of annoying if it's successful the response is actually going to be blank so that's good um upload file response it's only going to actually give us any information if there's an error so the ideal situation is this response is blank so um I think that's it so this using this example and so let's open our terminal and let's give this a shot um okay so I'm already navigated so I'm going to do python3 main.py and let's see what happens the main method upload file response and none so that's great so there's no error let's go to our S3 and let's find this bucket FP test one and we see we have this test file um that was uploaded just right now and let me click on that let me download it and open it it opens here here's my file I want to upload okay so this is working exactly correct exactly how we expected it so again just to recap we created an S3 bucket named FB test bucket one we then created an IAM policy which gave access to this uh which gave access to that bucket so our policy this FDA test bucket policy looking at that it gave access to that bucket then we created a user that we could assign that policy to and then with that user we were able to get credentials um so you know that user FB test bucket user we we generated an access key and a secret key that we were that we use in our script in order to you know programmatically access our AWS Services if I try to access any other services besides S3 that would fail because again I only right now this user only has access to that specific bucket which is what we want you know we want to try to be as minimal as possible with our permissions we you know we only want to assign permissions that are absolutely necessary which is what we did in this case and then um because we had access to that bucket we were able to use this um this photo 3 python package which is the AWS package to upload our file so um so yeah I mean this is obviously a very basic example there are a lot of things you could do with this um again what you know typically my biggest use case is I'm going to generate a log in all of my scripts and I'm going to take that log and I'm going to upload it into S3 and this way I'm able to just review those in the future like if there's any issue with any script that I have running you know if something doesn't run the way I'm expecting it to I can go and look at that log and that's ns3 and um I can go into S3 I can download that log and I can see what's what's going on there but obviously there's a ton of use cases which is why again I just wanted to make a very generic video not Financial related so hope you guys learned something appreciate you watching and I'll see you guys next time bye

Channel: Financial Python

Views: 952

Keywords:

Id: ACmQGfCzjkc

Length: 17min 49sec (1069 seconds)

Published: Mon Jul 17 2023