When to use an AWS S3 VPC endpoint

hi it's Tom Gregory here and welcome to this tutorial video about when to use the AWS s3 VPC endpoint and specifically today we're going to be looking at what problem the s3 vbc endpoint solves along with a working example of how to set it up and solve that problem so let's get right into it so to understand what an s3 VPC endpoint is we need to understand what problem it solves so imagine we want to get access to s3 from an AWS resource and in this example we have an ec2 instance that needs to copy a file from an s3 bucket and in this case it works because the ec2 instance is in a public subnet so has access to the Internet and therefore the ec2 instance can reach the AWS s3 URL to copy the file from the s3 bucket where things start to fall down though is where we need to access s3 from an ec2 instance in a private subnet as in this example and this doesn't work because the ec2 instance is in a private subnet so has no internet access and therefore the ec2 instance can't reach the AWS s3 URL and the request will eventually timeout an s3 VPC endpoint solves this problem by providing a way for an s3 request to be routed through to the Amazon s3 service without having to connect to subnet to an Internet gateway the s3 VPC endpoint is what's known as a gateway endpoint it works by adding an entry to the route table of a subnet forwarding s3 traffic to the s3 VPC endpoint this image shows a route table which has the s3 endpoint included we have a route that routes requests for the destination of s3 region Amazon AWS comm to the target VPC endpoint therefore any s3 requests will be routed through to s3 successfully in order to be able to demonstrate the s3v PC endpoint working and solving the problem that's just been highlighted we'll set up the following AWS resources and feel free to follow along with this first up we're going to need a public and private subnet in a V PC secondly we'll need in ec2 instance in both subnets so one in the public and one in the private subnet and we need an ec2 instance in the public subnet so we can access the one in the private subnet or in other words we'll be using it as a bastion and thirdly the ec2 instances will have an iron role associated with them that will allow s3 access and with this setup we're going to SSH into the ec2 instance in the private subnet and see that we fail to make a connection using the AWS command-line interface to s3 we won't be able to access s3 because we won't have access to the s3 service and then we're going to add the s3 VPC endpoint and then we're going to make the same request to s3 using the AWS CLI and see that it succeeds I'm logged into the AWS console here and first up we need to make sure that we have a public and private subnet inside of EPC so under services here let's go down to networking and select V PC I have a V PC called Tom V PC that we're going to be using for this example and if I go to subnets and if I search for Tom VPC I've now got the subnets that are in this specific VPC and we've got two we've got a public subnet here and a private subnet and it's worth pointing out the difference between these two is that if we have a look in the root table the public subnet has a root here to an Internet gateway whereas the private subnet doesn't so that's really the only difference next up we're going to create an I am role for our ec2 instances and this is going to allow them to use the s3 service so let's go to services again and this time go down to security and click I am and on the left-hand side here we're going to go to roles great role select AWS service and ec2 under common use cases and go to next and under permissions here we're going to search for s3 and select Amazon s3 full access and go to tags and then skip through again and I'm just going to call this role s3 full access and then create role and then here the bottom here is my new role so next up we're going to create two ec2 instances one in the public and one in the private subnet so once again let's go to services and under compute select ec2 and then go to instances and first up let's launch an instance in the public subnet and we can just select Amazon Linux to the first entry here and because we don't need much compute power we can just leave it as the default t2 micro and then under instance details we need to make sure we're putting this instance into the public subnet so I'm going to select my V PC and I'm going to select my public subnet and importantly under I am role here we need to select s3 full access and then next and next again and I'm just going to add a tag here and it's going to be a name tag and I'm gonna say public instance click Next are now under security groups I'm going to leave the default settings or allow access on port 22 for SSH access and then review and launch and then launch and I'm just going to select my existing key pair here and now if I go back to ec2 and under instances I can see that this instance is currently launching let's go ahead now and launch instance into the private subnet which is almost exactly the same process same ami image same instance type and under details here this time I'm going to select my V PC and the private subnet and then the same s3 full access I am role next next and then in tags I'm going to add a tag and this time we're going to say private instance to identify this instance default security group and then launch this with the same key pair now back in ec2 here under running instances we just need to wait for them both to start up and now that they've both started up I'm just going to double check I've got my public instance which down here we can see the subnet is Thom public subnet and then my private instance is Tom private subnet and first things first we're going to want to SSH into these instances and we're going to use the public instance as a bastion as a way to jump through to the private instance which doesn't have access to the internet so I'm going to copy the public IP here of the public instance now in a bash terminal I want to run SSH ad passing in the key of the key pair that I have assigned to my instances and that's going to add the identity to SSH and now I am going to run SSH - a + and - a is going to forward our identity on to the host that we go to so that we can then jump to another host so SSH - a of ec2 user which is the default user and then that's the public IP of the public instance and now I'm on the public instance I'm going to jump back to AWS and grab the private IP of the private instance here and we're going to SSH - easy to use it at that private IP and now we're on the private instance in the private subnet what we want to try out is to do an AWS CLI command and we're going to run AWS s3 LS which is a very basic command that just tells you what buckets you have an s3 and we need to specify a region and that's the same region as the instances that I've added here and we'll see what happens and right here this request is hanging and we'll see after well quite a lot of time that eventually times out [Music] and now after five minutes we've actually got the timeout I just wanted to show you that just to show you that genuinely is the problem here and of course we do have an iam role set up to make it so that this ec2 instance does have permission to talk to the s3 service but we don't have any internet access so there's no way for this ec2 instance to reach out and connect with the AWS s3 service so now let's go ahead and fix that by creating the s3 VPC endpoint so back in the AWS console here we're going to go back to services and on the networking VPC again and this time on the left hand side let's select endpoints and let's hit the big blue button here create endpoint and we're gonna search in this list for s3 and let's select the s3 endpoint and by the way the VPC endpoint is specific to your region and then we're going to choose our V PC and then we need to select a root table and the V PC endpoint will automatically update our root table to have a root to the endpoint and we need to select which one it should update and of course we need to select the one that is associated with our private subnet so the one over here which says Tom private subnet and down here we have the option to provide a custom policy or we can just say full access which means that the VPC endpoint will allow full s3 access to any AWS account and then select create endpoint and that successfully created and if we go back to VP see down here and if we go to route tables on the left here and select the private route table and this will take some time for AWS to update the route to include the VPC endpoint so let's just hold on a minute again and now after just a moment we can see we've got an additional route down here and if I just pull this to the left you can see that it's active and our destination is s3 dot region Amazon AWS comm and the target so when we try to access this euro the target is going to be our VPC endpoint here so now that's all set up we can go back to our bash terminal and I'm going to run the same command again awesome so this time it completes and it's listed out the s3 buckets that I have in my region EU West one here so what's happened is that I've added the s3 VPC endpoint and now we have the option from our ec2 instance to reach out directly to the AWS s3 service and if you're thinking to yourself why not just set up a NAT gateway in the public subnet as a way to allow internet access from the private subnet because this could be a way to be able to access s3 then there are two key points to take into consideration and the first one is that of course than that gateway it's not free whereas the s3 VPC endpoint is and the second one is that from a security point of view it's a really nice thing that you can keep your private subnet without the internet access and just use the VPC endpoint to be specifically giving access to the s3 service thanks a lot for watching and hopefully that's helped answer the question of when to use an awl s3v ecn point and if you want to hear about other interesting topics in the future then please subscribe to my channel and if you did get value from this video that please hit the like button otherwise I look forward to seeing you on the next episode of Tom Gregory Tech

Channel: Tom Gregory Tech

Views: 7,231

Keywords:

Id: uvKWJ4c1EYc

Length: 12min 40sec (760 seconds)

Published: Mon Apr 13 2020