AWS Pi Week 2021: Advanced networking with Amazon S3 and AWS PrivateLink | AWS Events

hello and welcome to advanced networking with amazon s3 and aws private link my name is spjundra and i'm a product manager in s3 in this session i will talk about vpc endpoints and private link and how you can use them to access s3 let's get started so i'll start with a brief introduction to vpc endpoints and how you can use them to access s3 from a vpc or from on-premises i'll do a side-by-side comparison with the gateway style and the interface style vpc endpoints for s3 and then we'll dive into some of the advanced use cases of interface endpoints i'll talk about the endpoint policies and how they provide you with additional layer of access controls in your network perimeter i'll share some of our partners and then towards the end i'll do a quick recap so vpc endpoints for s3 are secure endpoints in your vpc that provides secure access to s3 over the amazon network there's some easy to configure highly reliable and highly secure endpoints and you can control access to them via endpoint policies you can control who can access the endpoints and what resources can be accessed from those endpoints there are two flavors of amazon s3 vpc endpoints back in 2015 we first launched the gateway style endpoints to allow access to s3 from a vpc and we recently launched private link based interface style vpc endpoints to allow access to scy private ips let's see how that works so before gateway endpoint if you had an application in a vpc that needs access to amazon s3 you would typically configure an internet gateway and now and allow access to s3 via the internet with the gateway style endpoint you can directly connect to amazon s3 from applications inside of epc over amazon network without going over the internet this allows you to simplify your network architecture since you don't have to configure internet gateway or nat devices in your vpc you can control access to the gateway endpoints while endpoint policies the traffic remains on the amazon network and does not route via the internet and then gateway endpoints are free gateway endpoints are highly popular amongst many of our customers to access s3 from an application in a vpc and when you create a gateway endpoint it looks like this in the ui so you configure your vpc and the routing table when you select the routing table a routing rule with destination s3 and target gateway endpoint is added in this example i have my default route table configured which is associated with all six subnets in myvpc or virtual private cloud so any application in those subnets would route to sv while the gateway endpoint and when you use gateway style endpoints in your client applications they use the standard s3 regional dns names that resolve to public ips you do not need to configure a dns resolver or update your client applications to access s3y the gateway endpoint it just works let's see how the new private link based interface endpoints work so the same example if you have application in a vpc that needs access to s3 you can configure an interface endpoint for s3 and provision an elastic network interface or an eni in your private subnet in an availability zone an eni is a logical networking unit that represents a virtual network card in your vpc you can have one or more across multiple subnets and availability zones to architect for high availability and each of those enis are provided private ips from your private subnet then applications routing to interface endpoints gets routed by one of the eni's over private link network to s3 here i have a screenshot of the interface style endpoint in the vpc management console notice the difference when you create an interface endpoint instead of configuring the routing table you are provisioning one or more enis across azs and when you use interface endpoints we provide endpoint specific dns names that can be dissolved by a public s3 dns domain to private ips off your vpc so then you don't need to configure a dns resolver to use interface endpoints for s3 there are two types of endpoint dns names first we provide a regional level endpoint dns name which consists of a vpce unique identifier the service name which is s3 the region in this example usc 1 and vpce.amazonaws.com and when you resolve these dns names you get all ip addresses of all enis in your subnets the second type of dns name is zonal endpoint specific dns name that in addition to the vpc unique identifier also consists of your availability zone and when you resolve this type of endpoint dns name you get the specific ip of the eni that is in that availability zone so the endpoint dns names give you the flexibility to route your applications while the regional endpoint across one or more enis or why a specific eni using the zonal endpoint specific dns names depending on your application architect architecture so let's do a quick comparison between the gateway style and the interface endpoints side by side so as we discussed the gateway style endpoints use the public ips of s3 by the regional s3 dns names and they work at the routing table where a new rule in your routing table with destination s3 and target gateway endpoint is added and because they work at the routing layer they're not accessible outside the vpc and thus do not currently support on-premises applications or cross-region requests they're free of charge and there is no fee to use gateway and points to access s3 interface endpoints on the other hand use private ips from your vpc to access s3 your applications are out via the dns entries at the regional level or at the eni level and since they have their own dns name and ip addresses they can be accessed outside the vpc and thus do support on premises applications and cross region requests the private link-based interface endpoints are a paid feature and you get built a data processing fee and an hourly fee so while many customers use gateway endpoints to access s3 from a vpc many other customers in life sciences finance and healthcare industry have told us that they want a few things to access s3 from on premises first they want private access where the traffic traverses on there an amazon network rather than the internet they want simple configuration uh which that and they do not want to configure public ips off s3 in their corporate firewalls or configured an internet gateway on premises or configure proxy infrastructure in their vpc they want the reliable performance that they expect over their direct connect connection and many customers also told us that they expect better network performance wired direct connect versus the internet as then they avoid the internet congestion i'm happy to share that we recently announced support for private link for s3 which allows you to get the private access why are the interface endpoints in a simple configuration over the aws direct connect let's see how that works so you have an application or applications on premises that need access to an amazon s3 bucket in an aws region why a private ips you would typically have a direct connect connection or a vpn which connects you on premises that work with a vpc inside aws and before private link you would typically configure a proxy infrastructure that consists of a load balancer multiple ec2 proxy instances across subnets and a gateway endpoint then applications on premises send the network traffic to the load balancer with which forwards it to one of the proxy instances which connect to s3 why the gateway endpoint while this allows you to access s3 with private ips from on premises the proxy infrastructure typically constrains performance and is complex to manage and scale now with private link-based interface endpoints you can simply replace your proxy infrastructure with highly scalable and highly secure interface endpoints for s3 this simplifies your network architecture since you no longer have to configure a proxy infrastructure allow public ips and corporate firewalls or configure an internet gateway device and many customers also told us that they want the flexibility of only allowing certain applications on premises to connect via private link well the endpoint specific dns names of the interface endpoints give you that flexibility to allow certain group of applications from on premises to access s3 wire private link while the rest of the applications access s3 the same way as before so so far we've talked about how you can use the gateway style endpoint to access s3 from a vpc and interface style endpoints to access s3 from on premises and in this next section i'll talk about some of the advanced use cases of using both endpoints together in the same vpc accessing applications on premises and across multiple vpcs to s3 and then we'll talk about how to use interface endpoints to allow cross-region access to s3 so going back to our example of how customers use proxy infrastructure in their vpc with the gateway endpoint to allow access to s3 privately from on-premises commonly there are other applications in the same vpc that also need access to s3 via the gateway endpoint a popular example is data lakes on s3 that require applications across the vpc and on premises to access s3 so instead of replacing your gateway endpoint you can simply add interface endpoint for the for s3 in the same vpc as the gateway endpoint then applications on premises use the standard s3 dns names to access s3 why the gateway endpoint and applications on premises use the endpoint specific dns names to access s3 wireless interface endpoints this allows you to optimize costs since you only use interface endpoints for on-premises traffic and now you can leverage your existing vpc configuration with direct connect since you no longer need to create a new vpc to host interface endpoints and allow access to s3 from on premises another popular use case is applications on premises and across multiple vpcs connected via transit gateway and direct connect and you have applications that need access to amazon athena amazon kinesis or amazon s3 and a common way to do that is using shared services vpc which hosts interface endpoints for these amazon services and this provides a centralized access where all applications across vpcs and on-premises use this shared services vpc to access athena and finesse's so before private link to access s3 in this architecture you would create gateway endpoints in each of the vpcs to access s3 while gateway endpoints are a great way to access sv from a vpc this means that you would manage multiple endpoint policies across multiple gateway endpoints you can now replace the gateway endpoints and add an interface endpoint in the same shared services vpc to provide access to s3 in the same centralized way as you would to athena and kinesis in this architecture this allows you to simplify your endpoint policy management since you only have to manage a single set of endpoint policies for allowing access to s3 versus managing multiple endpoint policies across multiple gateway endpoints now let's see how you can access s3 from an application that is in a different aws region so in this example i have an application in vpca in usc 1 region that needs access to amazon s3 in amazon s3 bucket in u.s east 2 region you would typically con configure another vpc where your bucket is and connect the two vpcs together via the vpc pairing connection next you will create an interface endpoint in the vpc in the same region as your bucket and your applications would then access your bucket in a different region via the endpoint specific dns names over vpc pairing connection so we've talked about different use cases and how you can use gateway endpoints and interface endpoints and and we've briefly touched on endpoint policies in this next section i will go deeper into what endpoint policies are and how you can use them in your network perimeter so endpoint policies are a powerful feature that work in conjunction with in policies and bucket policies to control which im users rows can connect to which s3 buckets or access points using that vpc endpoint and each endpoint policy is attached directly to a vpc endpoint so you can have different endpoint policies across different different vpc endpoints to give different levels of access control to multiple teams note that vpc endpoint policies do not provide access to bucket by themselves you must also configure im or bucket policies to allow access to your buckets from the vpc endpoint i'm happy to share that we recently announced a new im condition key called s3 resource account that allows you to give access to buckets in specific accounts this is especially useful for customers that have multiple buckets across multiple accounts and now they can just simply allow access to their buckets in specific accounts instead of listing individual buckets in their endpoint policy let's see how this works so you have your own network that consists of applications on premises and in the vpc connected via direct connect you can use endpoint policies with interface style endpoints to extend the access control measures from your vpc to on-premises to control which im users can access this endpoint you can control only my im users in my organization and in my network and access this endpoint and anyone else will be rejected or denied access you can also configure endpoint policies to only allow access to your buckets and then that would restrict access to any bucket that is not in your organization or not in your account in this example we'll use the recently launched sd resource account condition key to deny access to anyone except if they are if they're accessing buckets in this specific account using this new condition key next you can also configure your bucket policies to only allow access from a specific vpc endpoint but any im user that is not using that vpc endpoint will be denied access and in this example we can use the source vpc-e im condition key to only allow access to my secure bucket from a specific vpc endpoint i am happy to share that our partners veritas commvault hashicorp rubric elastics snowflake and the sunni support private link for s3 so if you have applications and workloads and backup recovery and data management that use one of our partner solutions you can connect to them using private link for s3 so we've covered a lot of ground let me quickly recap so we talked about how vpc endpoints allow you to connect to s3 without going over the internet without the need to configure an internet gateway or nat devices in your vpc and how you can simplify your network configuration to access s3 from a vpc wire the gateway endpoint and on premises by the interface style endpoints using private ips from your vpc we delve into some of the advanced use cases of how you can use both end points on the same vpc to optimize cost and only use interface endpoints for on-premises traffic and use the gateway endpoint for nvpc traffic to s3 we talked about how you can use interface endpoints in a shared services vpc to provide the same centralized access to s3 as you did to other aws services like athena or kinesis we also looked at how you can use interface endpoints to allow cross region access from applications to a bucket in a different aws region via vpc creating and interface endpoints vpc endpoints are building blocks that can be used separately or together to fit your use case and finally we talked about endpoint policies that can be used to control access in your network perimeter you can control who can access the endpoint and what resources can be accessed from that endpoint that concludes our session thank you so much for your time all right thank you very much piace uh you know i i learned a lot i i really appreciate the slides where you were showing how complicated it is to kind of set up your own proxies manage all that infrastructure you know and let's face it even that was already an improvement before any of that existed right if you think about the legacy workflows trying to move data across data centers in the cloud doing so securely this is just a huge challenge and enterprises struggle with this all the time no sorry i shouldn't just say enterprises but lots of customers struggle with this all the time and i'm really glad to see the kind of incremental improvements and the journey that you've been able to take us through you know um as you were doing that you know maybe maybe i'm kind of curious just as i was watching the talk like what what was the aha moment when you took a look at something like hey um do we need something like private link or is direct connect sufficient what what was the you know was it just kind of like hey enough customers kind of gave you that feedback where you identified it as a theme or how how did the team kind of synthesize that feedback into what needed to be built next yeah the the basic pillars that you know we can categorize the feedback is first they want private access right so with the interface endpoints you streamline and simplify your architecture to get private access without using public ips of s3 in your corporate firewalls and many customers asked us to provide that direct access from direct connect so since the end point the interface style endpoints have their own dns names and ip address you can directly access from on-premises and third the the endpoint policies is really powerful and using them with interface style endpoints customers are now able to extend that control measures from their vpc in their on-premises network i think becky talked about this too but that that's been a another big feedback that we've been receiving from customers

Channel: AWS Events

Views: 1,378

Keywords: AWS, Events, Webinars, Amazon Web Services, Cloud, AWS Cloud, Cloud Computing, amazon s3, cloud storage, object storage, data lake, data storage, 15th birthday, 15th anniversary, pi day, pie day, pi week, pie week, birth of the AWS Cloud, simple storage service, s3, aws s3, aws pi week 2021, networking, privatelink

Id: m3ngS7IG8Ls

Length: 20min 39sec (1239 seconds)

Published: Tue May 11 2021