Setting Up AWS Transfer SFTP - Step by Step

hey what up y'all what up what up i am blue and today we're going to be talking about aws transfer family so we're going to kind of walk through the process we're going to kind of discuss what what is it what is transfer family uh we're gonna discuss on how to set it up and some of the the uh the set the settings that need to occur before you even try to set it up like making sure you have a policy created and aws i am make sure that you have a a row created uh we'll walk through the process of creating some ssh keys that we will need and so we'll kind of go through all of the prequisitions of what you need to do before you even try to set up your aws transfer and then we'll set it up and then we'll kind of go through some scenarios on why this is beneficial on why this could be a replacement to your old traditional way of um managing files communicating files by sftp so we'll kind of go through all of that but let's go ahead and get started guys hey the sponsor of today's video is go share it go check them out at go share it dot io it's a sharing platform that allows you to share large and small files where it makes it very easy to share your recipient all they need is a download link and they're good to go again go check them out also guys before we get started you can give me a like give me a follow i appreciate it you know i've been getting a lot of lot of support uh appreciate for the followers and uh let's get started guys so let's kind of review real quick what is aws transfer why would you even want to set one up what's the purpose behind it and all of that right so anybody who has communicated in the past with like an ftp or sftp site you know just the basic mechanics of what that is is pretty much you have a server you set up again sftp or ftp protocol and that is a method of transferring files right you normally will use a tool like filezilla um there's other tools out there but you know you'll use some kind of ftp tool and which allows you to just drop files in there and ultimately it'll upload it to the sftp or ftp server and then or you can download files as well right connect to it through to the client to the client application like filezilla then you could just download files as well so just a method of being able to communicate files a lot of businesses again use ftp sftp for the purpose of communicating files um you know so a lot of the one thing you hear a lot in programming is like apis well if you're not dealing with api if you're not communicating data that way most likely you're going to be communicating data through some sort of method like an sftp um where you're handing off a file then you're receiving a file right it kind of goes both ways so and and that's where this comes in now a traditional the what i want to call you the more the traditional way of doing it would be um if if you're dealing with a client and then let's say they're giving you files for you to process right you're going to do some etl processing and ultimately load it to the data warehouse and doing all that kind of stuff the way it would normally work is they would potentially just drop it to an ftp sftp site and then you would have to have a process to connect to the sftp download the file then ultimately let's say move it to an s3 bucket or move it to an azure storage or somewhere right but you're not moving it and norm again i'm talking about more like cloud services right now but you will move it somewhere else in order for you to process it to do something with the files well one of the things that they do amazon did here with aws transfer family pretty much what it is you you're setting up a server and a service that is pretty much is going to act no different than a than a traditional sftp site you know or ftp and the only difference is the backend the backend instead of you having a physical server where the files are being dropped into and then you have to ultimately access those files either by connecting to the physical server or by connecting to the to the ftp or smtp site to download it in this case as any as files are dropped into the sftp or ftp they're actually being uploaded directly to an s3 bucket which makes a big difference because if you're processing files then now you can start automating a lot of the processes and files get dropped into your sftp using aws transfer family it'll take that file save it directly to an s3 bucket and then from there you could have either triggered you have schedules running that run every hour you have triggers created using aws lambda where it's kind of like uh anytime a new file is dropped into your s3 bucket it will execute a process which could be an etl process whatever it is again the whatever you could do after that's unlimited but that's a key aspect of it right and what that also does it kind of gives you some flexibility where if you're trying to automate certain processes that maybe you don't necessarily want to write code to do for different reasons there's some organizations where maybe you don't want to write some code to do a specific task because maybe you don't have programmers or you know a lot of programs available internally so you want to kind of have more like some kind of tool to be able to at least build it out and doing something like this comes in very handy i have another video later that i'm going to make talking about how we're going to pull data from sharepoint and then we're going to upload it to this sftp site that ultimately going to drop it to an s3 bucket and we're going to do all of that without writing any code we're going to use microsoft power automate to make that happen so that's that that's a video that's going to be coming soon but again hopefully again that kind of gives you somewhat of an understanding of what it is that's what's called file transfer or transfer family you're you know pretty much you're just moving files around but in this case it's moving it directly to an s3 bucket now it does have for the efs that's not what we're going to be talking about here we're talking about s3 so that's kind of where we're going to be dropping files in there which again once you do that you're in pretty much you're lim unlimited on what you could do you could have all kinds of stuff happen automated after the fact to streamline the process any kind of process you want and you can see down here in the diagram where you have your um you know existing users that would drop again they will use some kind of ftp application like filezilla or so on drop files ultimately it'll drop into this bucket then from here you could you know again have an aws glue job run uh aws lambda job running um i mean the list goes on and on on what you can do with it so uh let's go ahead and get started so we're gonna kind of walk through the very basic stuff what needs to occur the first first thing that we need to do is i'm gonna go to my im manager management console we're gonna create a policy so the first thing we need to do is create a policy the policy that we're gonna create is gonna have it's gonna specify um well you know what before we create a policy i actually just take one step back let's go to our s3 bucket and what i'm going to do is i'm going to create a new bucket in here so let me create a new bucket this new bucket is gonna be specifically for sftp so i'm gonna call it like sftp give it some kind of unique name whatever name you want in my case i'm gonna call it sftp um i am lou oops i am lou so that's going to be my name smtp i am blue um must not contain uppercase letters uh let me get that fixed so sftp so smtp i am lou that's gonna be my name i don't think it's tooken yet so it should be fine leave the settings as is that's fine create bucket cool now i have a bucket in here called smtp i am blue so what i'm going to do here and we're going to walk through the process so just think of it like any sftp that you may have in your organization you may want to share access with multiple clients but you don't want clients you have access to the same folder so let's say for example i'm going to create a folder right now and i'm going to call this um client a oops client let's call it client [Music] um [Music] client a that's this folder then let's create another folder we're gonna call this client b right because again you can have multiple clients and now we're going to utilize this one bucket to be able to create subfolders inside where some objects pretty much but either way subfolders that's how we see it as um so the client that is given access would not um will not see their uh the other people's information they want to see their information pretty much okay so now that we have that set up let's go ahead and now go back to our i am console we're going to create a policy the policy is pretty much saying they have access to be able to access the s3 bucket environment to be able to do puts gets deletes and so on so let me go to json and i kind of have that already set so let me just copy and paste it in here now can i explain what you know just what it's just doing so as you could tell the action is gonna this case s3 listing it's being allowed and it's going to be for the bucket that i created so in my case i'll call it sftp i am blue that's my bucket so this is what this policy is going to have access to so why am i creating a policy in this manner i'm doing it because i want to live i want to be very specific of what they're able to access um for example let's say i may create another bucket for another group of customers that are going to have access to an sftp for whatever reason right and in that case i could create another policy and things will be separated they will not uh you know the from a policy standpoint one policy will not have access to all the buckets so it just kind of narrows down your from uh just kind of keep things separated and you can see we have um access to put objects get objects delete objects get put all of that so it gives us pretty much what we need uh next tag no tags review [Music] i'm going to call this sftp transfer that's the name of my policy sftp transfer right uh boom and it tells me s3 limited rewrites permissions cool let's create policy all right so now that my policy is done smtp transfer so keep that in mind because we're gonna use this uh later um now let's go to rows so in the rows i'm going to create a new row this is going to be aws service that's fine uh let's go next oops um this would be let's see this is going to be transfer now let's hit next oops transfer and next so in here this here we have our policy that we just created right now right sftp transfer we want to attach this policy to this row so let's go ahead and click on it we're going to attach that hit next and then this is going to be let's give it a row name so what what am i going to call this i'm going to call this uh sftp [Music] transfer let's call it sftp transfer that's going to be my role name uh trusted entity so down here on the trusted entity it's going to be this is valid yep yep this is fine this all looks good so i'm going to hit create row cool so now if i go to my smtp transfer row we have our policy attached our trusted relationship and giving us trust to the transfer service so now we have access to two services really to the s3 bucket but that's very specific bucket then the transfer service so those are the two things that we need which we have access to so now that we got this created there's one more thing we need to do which is going to be we need to create our ssh keys because we're going to create an sftp not ftp so the difference between ftp and sftp is ftp is not encrypted so the only thing that's required is to put in a password to log in but as data is transmitted over from a client s ftp client to the actual server on the data not being encrypted or anything either sftp is and it required a specific key to be able to access the sftp so it kind of adds some some layers of security but we're going to be doing sftp which is why we need to create a key so there's two ways to create a key i'm going to walk through the process of the first way to create a key one way would be we're going to use putty to do so so use when sap if you have one sap installed so this will be more from the my windows users who are using windows um use that because that's most likely what you're using go to tools run puttygen make sure this is at 2048 uh ford yeah 48 rsa then we're gonna click generate cool so one of the things i want you to keep in mind we're not going to put up paraphrase for now we'll leave that blank um this key public key here that that we have right here this is a open ssh format key all right um and then of course um if we need to create a private key that's going to be open ssh then you will go to gen export open ssh key that will gener generate create a private essay is open ssh key format so by default if if i were to save this like save public and save private it's going to save it under the ppk format pretty much it's when sap format the ppk format it's a different format than the open ssh format i'm not going to go due to too much detail into that but most applications like um again later on i'm gonna i'm gonna be making a video of using microsoft power automate and connect it to an sftp site in that example um and that you know in order to connect it it's not it's it does not like the ppk format for your private key it actually wants the open ssh key format so most applications are going to want open ssh now if you're using putty of course to connect to this server to be able to download and access files and you know and so on then that's where you want to save your your public and private key in the um the format that you know putty generates for you and in this case though we want the open ssh key format right so just kind of keep that in mind this is the format that that i wanted in so what i'm going to do here is i could take this guy let me go to my text editor and then i'm gonna save it and this will be my public key so i'm going to save it somewhere let me save it i think i just created a random folder file somewhere so i just want to call this public let me call it public um open ssh um that's it i don't need an extension doesn't matter save next i'm going to do export to open ssh key yep no paraphrase that's fine this is going to be my private key right keep that in mind private open ssh um [Music] let me save so now that it saved it now let's let's just let's take a look at it real quick right they just so here's my you could tell you're seeing the header which says private and then this one doesn't say but this is public this public we're gonna have to assign this we're gonna do that in a minute we're going to assign it to the user that we're going to create that could have access to the sftp site so again this is one way to do it using putty so keep that in mind the other way to to do it and in my mind this is going to be the more easier way to do it and let's kind of walk through that process go to your terminal uh where is that don't have it open go to your terminal um we're going to do ssh what is it called the key gen so right by default is going to save it to my my user profile folder and that's fine just save it there then you can move the files over it is going to save it under the id rsa file format file name which is fine we could always change it nope paraphrase now you could tell in my mind this is way more easier to do it then with putty you gotta open up the app do this that whatever then x you know export out open ssh this is very straightforward if i go now to [Music] uh user ssh these are the two files that i have so let's let's open it up and let's take a look at it this should be our private key as you could tell it looks very similar to this private key um slightly different but very very similar right but over you know um if we take a look at our public key the public key looks very similar to our public key over here same thing right a little bit longer but the same you know it starts off with ssh rsa same thing ssh rsa then at the very end ends with s rsa key that has the date year months over here does they have that nope this one has something different it has the name of the computer that was generated under so because they kind of use that information to kind of generate the key so either way there's two ways to do it i prefer the um the terminal way to do it nevertheless though if you're using windows if you're used to putty then just go to puttygen and then just generate it and then once you do that go ahead and this will be your public key in the s open ssh format which i strongly recommend to use because that's what most applications use anyways and then just do an export open ssh to export out your private key as well just save it as private all right so now that we have our keys generated now we can get started so let's go ahead and get started now so now i'm going to go back to my aws web services we have again we create a policy we create a row and we have our keys so those are the first three three things that we have to do before we even get started with aws transfer right so let me come back i'm gonna go ahead and delete this i don't want this anymore this is the one i was testing with so i'm gonna stop let me stop uh let's wait to finish here yup it's stopping so do keep in mind with aws transfer family it's not like an ec2 instance where when you're not using it you could shut it off like stop it and you don't get charged that's not how it works even if you stop it you're still going to get billed it's one of those services that once you generate it's there it's either running uh i mean i know i know give this two option to stop but i think the stop option is probably more think of it more of a um your end user to have access to it that your clients that you gave keys to you to be able to drop files into or download files they just cannot access it anymore but the service it's still you're still being charged it's still there and so yeah just keep that in mind so that it's it's not like ec2 where you at least you stop it you don't get charged anymore that's not how it works i believe the price it's i mean it's like a hundred and something dollars as well per month that you're gonna have to pay so just kind of keep that in mind as well when you're thinking of if you go down this path to to manage it so now let's go ahead and create a new server okay so now we're starting from the beginning let's set up an aws transfer right i'm going to create new server i'm doing sftp which is why i'm going to have that clicked on you can do ftp if you want but that's not what i'm doing i'm doing sftp uh service manage you know so that's what i'm using um i'm not using the aws directory service if you are then you can use that but in my case i am not um so i'm doing service managed i want this to be accessed publicly because again if i'm going to be giving clients access to drop files then it needs to be publicly available not just internally where people that already have access to the vpc uh i'm not gonna provide a custom hosting and you can you can set up go to routing 53 to set up the dns i'm not doing that just keeping it basic keeping it simple um let's see let's click on next i want this to get dropped into an aws s3 bucket so that's why i'm got that set i'm not going to create a new role because remember we created rows already in the right before that we created a policy a row and our our keys since i already created a a row let me go ahead and choose the row and it's let me refresh there goes sftp transfer so that's my new row that i created this is so these are the different um algorithm policy that it has leave it leave it by default i believe 20 2006 should be fine i did some testing with the 2022 and i had issues with it so but i think 202006 should be fine uh that should work you should be okay with that um this could stay blank that's fine this is more of you're moving from another server we're not doing that like an existing service already that you have this is a brand new one that we're creating so we're not going to do that you could add your tags of course if you're going to have many what what's their what is it for the purpose maybe you got a test and a production that's going on where you you could add your your tags right maybe you do something like service type and this could be production then you may have another one called test you could do something like that but either way it's up to you manage your workflow we're going to leave that blank we don't need that right now banner no banner i'm not providing any banner in this case click on x um yup everything should be good let's create server so it doesn't have any users which is fine right because we just created it so we'll get those new users added so as this is being created keep in mind remember we created two folders in here these are going to be our two users that we're going to create one user again we call client a the other user client b the way i would normally do this like fill this with production if um and they just do this for the purpose of testing right let's say my client was called my first one of my clients was apple right so i will use their name as the folder let's say my second client was called google right so these are two my two or your clients let me go ahead and delete these guys cool so now that we got that deleted we go back now we have folders inside our s3 bucket that are client specific so when we get ready let's kind of go back let's see if this is done nope it's still starting up so whenever this is started up what we're going to do we're going to add two users and we're going to create those viewers we could call the username whatever you want but each user is going to be pointed to a specific let's say we call the first user um i don't know just apple um apple user apple or something generic right then we call the second user you their google well each user is going to be pointed to a specific folder in our s3 bucket so as they connect and upload files or they connect to be able to download files they would only see files that's in their folder only so let's see if this is done uh it's still starting it takes a couple of minutes so hopefully it gets started it finishes in a minute going let's try again nope still going let's try again still going let's try again finally online got it all right cool so now that we're online that means that the server has started again tell the new users so we're going to add users let's go ahead and um click on these the server we'll see that it's a sftp protocol all of this is fine again this is our endpoint this is what we're going to be using if we had a custom host name then it would be something like sftp dot i don't know uh myservices.com or you know whatever domain you have that's kind of the way it would be if you we set it up that way if not this is what we will provide the the client in order to connect to the sftp site so we'll we'll walk through that in a minute so this is our user section that we have here right so one of this is we're going to create users let me go back to my file so did i move these files over i don't think i did i want to copy these so remember how we created two private two keys one using putty one using the terminal so we're going to need two keys because one is going to be for one client the other one will be for another client so let's go ahead and actually rename these right i think then we call our client apple one called apple so they just name the files properly so we know which keys belong to which client so that that would be apple the other one would be google private this is a private uh i don't need to put open ssh to leave it blank then this other one will be our public key this would be google public cool so we have our keys for apple and again this will be just the name of your customers right so you want to create keys for each one which is what we did um okay so now we're gonna do add user our first user we're gonna call this user um oops we're going to call this user apple right let me just call it um let me call it like sftp [Music] apple and that's the name of the user the row that we're going to use is going to be the sftp transfer row because around we create a policy in a row so that's what this is um then we're going to specify in our home directory this is going to be our bucket so in this case it's the sftp i am blue bucket and then we're going to this section here is going to be which folder they could have access to which is gonna be the apple folder and we're gonna put restricted which means they're straight they're they're restricted to that folder only they cannot see nothing else right so kind of tell the little details on the side next we need to copy and paste the public key in here so where's my uh let's close out of these um pry apple public key [Music] so i'm going to now copy and paste my public key boom paste it in there that's it tags i'm gonna leave it blank then we're gonna do add so now i'm gonna add another one another user i'm gonna call this user sftp [Music] google the row is going to be same thing sftp transfer this will be the sftp imdrow and then the folder they could have access to will be google and it will be restricted as well now let's go ahead and get our public key for google which is this guy and then i'm going to copy this over then we're gonna hit add so now we added two users to this right um apple and google in your case you call them whatever your clients are or whatever you whoever you want to have access if i go back to our bucket let's let's look at apple it's blank there's no files in here nothing so now i'm gonna go back to my when sap and we're gonna create a new connection so again let's go back to this is going to be my endpoint is going to be my host so that's my host port 22 it's fine because it's sftp then my username for apple is called sftp apple that's what i called it right now there is no password because we're using keys to to authenticate so that's going to stay blank in order for me to provide my key um i would go to advanced authentication and then it will be private key um now the only thing i just realized here i don't i don't think we created a ppk file which i think that is what it's looking for since i'm going to be connecting with putty i should have created one so let me go ahead and create one right quick because i'm connecting in the manner that i am so let me go to run jen i'm going to load my private key [Music] then i'm gonna save my private key and i'll save this as um apple private up yeah just apple private should be fine let me go back to the folder yep it's a ppk file so they're the extension so now let me go back i don't need this i'm going to go to advanced authenticate private key here's my apple private key in the ppk format i'm going to hit okay let's hit save for now just say we need to come back make changes i'm going to call this ftp apple hit okay so now let's go and click on it yes i want to get this cache boom i'm connected so now that i'm connected i'm going to drop a file in there let's let's go ahead and drop a few files i'm going to drop brave my aws command line let's drop these files in here so far it looks good it's uploading skip this has to do with like time stamps and i don't care about that so i'm skipping it so now that i uploaded it if i go to my s3 bucket hit refresh boom we have data in there now again keep in mind if i go to my um my client the highest level that i could go to i can't even move up a level see so the only i don't even see a folder all i see is what i see now i believe once i'm in here i can create new folders if i want like if i want to create a new directory let's call it test um hit ok if i come back over here we'll see a new folder here called test so they have the access to do that but the only level they could see is whatever's inside this apple folder that's it you cannot see nothing else you know same thing goes with if i were to connect with um the others another session right i'm going to create a new session and this is going to be for the other account the other user that i created which would be uh sftp [Music] google we're gonna go to advanced um our private key let me i'm just let me test it out real quick i think i'm one um normally i always open it up with a ppk i'm wondering if it worked with the with the open ssh private key format uh nope there you go they asked me do you want to convert it so it does it doesn't need to be converted to the putty format which is a ppk format so i'm gonna say cancel cancel what i'm gonna do go back to run gen load i'm gonna load my private key but before i do that i'm going to just let me rename this to open ssh format so i'm going to reload this guy then i'm gonna do save private and then i'm gonna call this google private that's it save cool that should be done so now let me go back go to advanced authenticate and then i got my google private but again this private happens to be in the ppk format which is the putty format uh so just keep that in mind if you're using putty to connect you're to need to convert it to the putty format i'm going to hit ok uh then save let's call it sftp google so now let's go ahead and connect so as you can see when we connect there is no files in here but we could go ahead and drop a few files let's drop three files in here we're gonna again skip the timestamp once we get that little that error message that pops up for time step skip all cool so now you we say we have files in there again if i go to my s3 bucket there's nothing in here because very specific if i go to google folder boom we have our our files in there it's separate so as clients connect just like the way i did they could only see their stuff and their stuff only that's it they don't see nothing else now as you can tell since these are these files are being dropped into an s3 bucket this is where you can get creative right to more streamline and automate your process so if you're if you're doing data processing for your clients and they have to provide files for you that's where now you can be like okay file or drop you could create a trigger to or have a schedule that will start extracting the data from these files let's say load it to redshift or manipulate the files or or whatever right it's really up to you um on what you're trying to do but this would help automate that process where before the the traditional way of doing things would have been uh traditional sftp they gotta drop it in there you gotta have an hour process to download those files whether it be periodically most likely but download the files and then you end up moving these files over to an s3 bucket or so on then the process starts now you kind of skip that step because the files being dropped directly to your s3 bucket and then you could just have a trigger to go off using lambda a schedule whatever right it's really up to you to do whatever you need to do so again i have another video coming up where we're going to use this and we're going to ultimately use power automate to have a schedule that's going to run that's going to download files from sharepoint and it's going to connect to this specific sftp that we just created which means it's gonna be dropping files directly to an s3 bucket so i'm gonna show that process of it and that what i just mentioned right now is a non-programmatic way to move files from sharepoint to s3 buckets without using any code that's pretty much what we're doing in that case so that video will be next that will be one of my next videos to come hopefully this helped out out hopefully this kind of gives you a better understanding of what is aws transfer how to use it how to set it up not just how to use it but also as you saw there were some prequisitions that need to be set up first like policy row um ssh keys you know you need to do all of that first and then you go to the process of setting it up but it's very easy just like as you notice when i went to add users but once you've got everything set up become very easy after that to to add new users to get them set up so if you're going to be giving um clients where a uh sftp access you know service where they you know you're the one hosting it this is a good solution and it's very easy to add new clients new users as you know as uh as needed but again guys hopefully this helps out give me a like give me a follow appreciate everything man i appreciate all the love i've been getting here on the platform and talk to y'all later

Channel: I am Lu

Views: 10,649

Keywords:

Id: Hg4H6eQWjq8

Length: 42min 25sec (2545 seconds)

Published: Mon May 30 2022