AWS re:Invent 2017: Networking Many VPCs: Transit and Shared Architectures (NET404)

all right wake up you guys that front of the party last night yeah looks like it so yeah really happen for all you guys to be here I'm excited I'm excited you guys came to learn about networking at 10:00 a.m. at after the party so I'm I'm honored by you guys so I'm Nick Matthews I'm a Solutions Architect I do a lot of to spend a lot of time with customers and our partners working on network architectures and things like that so today we'll be talking about connecting mini V pcs we've also got Vicente up here he's going to join us later and talk a little about his use case so let's get into it man this talk is about many V pcs so what does that mean you know when I talk to customers like you know one of the usually one of the first questions I have to ask is like you know how many V pcs do you have sometimes it's one so if I'm just three what do think you're gonna be in a year is it 20s and 30s a 500 years a thousand because you might start off with 10 then you have 20 and you figure how that goes and then before you know it you got hundreds and you know it's sort of an interesting problem to go try to work on and try to solve so that's that's the that's the topic for today is like you got a lot of BBC's what do you do so let's start off with virtual private cloud VPC hopefully you guys know this is a foreigner level session so this is my only teaching slide but they say a virtual private cloud is your closely resembles your traditional network and we do it in the cloud right so that's that's mostly true so if you know I've been doing a lot of networked engineering a lot of customer orkut Xers those kinds of things this is what pretty standard sort of premises network looks like so you might have two data centers you might have sort of you know fibre between them you apply you have some sort of wide area network or when maybe do it some MPLS but it's VPN going on so if you turn this into V PC and 80s stuff it mostly fits right so we can still do VPN the the private fibre sort of turns into V PC peering we'd put things into instances we connect the LAN in through Atos Direct Connect and this is a mostly one-to-one sort of equivalency but there are some differences so what are some of the differences with VP C's so we say it's resembles a you know traditional data center but how many you guys created a you know a data center in seconds it doesn't happen right you can go into the console and just keep clicking and you've now you've got like ten data centers like that doesn't happen on premises although if you do like we probably ought to hire you so it's really easy to create VP C's another component is the access models so back in a prior life I used to walk around in data centers I had my Cisco blue console cable and you know I could probably plug into customers routers and do stuff I would never do that because it's totally wrong but I could probably do that but there was inside that datacenter there was not like a single access credential that got me access to everything in the data center like I'm pretty sure that mostly because those people don't talk to each other very well it's like the storage people on or whatever but there was no single credential that would get you access to everything in that data center and a V PC you have a root account the root account has access to everything that's why we had to be very very careful with that root account and so that's a difference one of the things is ownership so who are my network people in the room pretty good portion but there's a lot of people that aren't network people that are creating networks so developers land business folks application people and they're coming and they're trying to figure out how networking works and so you may end up with a scenario where those guys just want their own network or end up creating their own networks and that's a difference as well because that typically doesn't happen on premises so you know one of the components here that we're going to talk about is mini V pcs and this usually comes up because there's a prior conversation at the customer right so there's this sort of divide that I see it's it's a gradient more than a binary sort of option but do I want to have really big accounts in V PCs and get sort of granular there or do want to have lots of lots of accounts with maybe their own V pcs and so there's there's a couple of things what we're gonna walk through real quick so if you do those sort of large approach you know there's less infrastructure there's less networks you know less accounts to set up you know obviously more on the other side whenever you start getting into controlling that environment you're going to talk a lot more about Identity and Access Management you know reducing roles making sure you know you're going to doing a lot of tagging you know resource restriction and controlling a lot of things within that VPC because you know it's sort of a big shared environment so those are some of the things you're gonna be looking at and the the case where you have lots of little V PCs or lots of accounts you know it's really more infrastructure related so it's more about VPNs Direct Connect you're probably one set up some standards so that like your public subnets and your private subnets have some sort of routing thing you've agreed to otherwise you can run in some trouble there so you may want to think about automation and standardization there the billing you know as a component here - right so in the one V PC the billing might be complex there might be multiple organizations using the same thing so it's be very careful the tags those kinds of things if you have lots of accounts you know each person has their own account that it's pretty easy to do that you know the billing so that's that may be one component and then we can just sort of the security and blast radius components which is you know what happens if that one account maybe the developer writes of for loop and just forgets to do it right and just floods the API right then your production people can't use the API either so then okay well let's put out development in production you get to this blast radius sort of scenario as well as some of the security concerns it's like I said like the root key scares people sometimes so you just be very careful with that and sometimes that's easier to manage with lots of accounts are lots of epcs so well the way I usually break this down for customers there's obviously a lot of sort of ins and outs here so what I what I usually sort of summarize this down is is if you if you are if you are a policy and sort of a you know access person the larger bbc's are probably closer to what you're going to be good at me I'm a network guy I'm an infrastructure guy so I'm totally biased here I spend a lot of time with infrastructure and network people so they they're comfortable that they like building lots of networks and that's just something they've they've got sort of culture of and they know how to do so they they sort of go on this orange path here of you know lots of accounts and those types of things so there's no there's no right or wrong here like I said it's a gradient of sort of decisions but if you end up going down the path of like one really large V PC this talks not we're gonna do much for you because it pretty much already covered all the stuff you need to know if you have one V PC so you know there's a lot of other sessions here this is sort of just for resource for reference most of these sessions have already happened but you know go and think about you know other approaches here there's some security concerns there's networking concerns a lot of ways to sort of think about this this V PC an account sort of approach so what we're going to do here is we're going to focus on a couple things one is you know we're really comparing things here because there's if you came here looking for like the magic answer there is no magic answer it's it's design right so design is always taking trade-offs figure out what's gonna be right for you and so that's really we're gonna you know plan on doing here we're focusing on a couple thing one scale right because we're thinking towards hundreds of EPCs towards tens of gigabytes and hundreds of gigabits of throughput at some point right what does that look like connecting V pcs and multiple sorts of scenarios and when we start thinking about scale we also anything about automation so what does automation look like so there's a couple different design patterns we're really gonna spend some time on we're gonna talk about the transit vbc you know I think every session I've gone to so far people come up to the front they go tell me how the transit UVC works or tell me about this so we're gonna get through a lot of that stuff today and there's two components to it there's sort of the transit BBC there's also a component with firewalls we're gonna talk about how what does this look like if you're using Direct Connect how does that impacted now we're going to talk about shared services models so if you have some sort of shared service you want to share it to mini V pcs what does that look like and then also we're gonna talk about multi region things so this is sort of like the blueprint of what we're gonna go down so let's get into it so this is sort of our starting point right up here we've got VPN we've got Direct Connect coming into our green environment where we've got a development and a production V PC so you know in this case just for the simplicity of talking about options I've put both PPN and Direct Connect up there you know but this would be a highly available network you know so this is this is sort of our starting point this is pretty simple this is where a lot of customers are at you know if you've got two VPN tunnels or to VPN connections which is actually four tunnels helpful Direct Connect you know connections in here and then I don't know then the blue group joins us the blue group they've got four V pcs so they've got maybe two lines of business now you know things are starting to look a little less clean you know there's a lot of connections here so you know I was talking with a customer this week where they said you know every time we add a VPN connection we have to open a ticket with a networking team and they just either out of laziness or contractual reasons that takes 45 days for them to do anything and so like you don't want to add like we're all about agility and scale and you don't want to wait 45 days for a VPN connection so that's and there's a lot of them too so some networking here's really like tunnels a lot of them don't and so you would like less tunnels in a lot of cases as well let's let's talk about connecting these things because because right now when these guys want to talk to each other it's going back on premises whether you're using a VPN or Direct Connect you know it's going to some other device just to do for these things to talk so the you know obviously vbc peering is a good idea there so let's connect let's just say we want to connect development production so well that's gonna look like is we're gonna have three vbc peering connections right but then they go yeah you know actually the whole blue environment really wants to talk to each other so what does that look like okay well that's just another four period connections still pretty simple so if you're in the sixty PC stage you know this still makes sense but what if we go beyond that what does it look like so we just we got the the pink guys they just came up and the blue guys add is more stuff so now we're now we're looking at other problems we've got a lot more things to add a lot more connections how is this how we scale these things and so the sort of main option that we're using right now for this problem is the transit V PC so the transit vbc is basically a centralization point where we've got some VPN instances and they do this or the east/west and all the connectivity for this so essentially once you add a V PC to the transit V PC has access to every other VPC right so what are some of the Bennett's here so essential ization is a component right les tunnels easier for me to go manage that that pair of devices also scale so VPN with its Goods and Bad's when the good part of VPN is it's really scalable you can do thousands of them and well they work everywhere VPN works over regions and accounts and different networks so it's it's highly scalable as well as you know is inter region so obviously we have had pretty exciting announcement this week about BBC peering in a region which yes that was exciting to be there for that announcement so that actually has reduced the requirement for transit BBC a bit and we'll get into like what that actually means but sometimes it's also about just being familiar you know like I've still got like Cisco commands like on my fingers in my mechanical brain that it just can't get out some people like continuing to do that so some people like what they know and that's you know one benefit as well we're gonna talk about security a little bit and how firewalls get inserted so transit we V PC tends to be a pretty dominant deployment model for firewalls and then also encryption we like encryption we like security and so it's a good way to do things like encrypt over Direct Connect and encryption everywhere so there's there's sort of two components we're gonna talk about around transit PPC the first one is the architecture so I'm gonna speak generally about how this thing is built what the sort of theories and why we made these decisions and then we're gonna go out later into the transit BBC automation that a OS built so we're gonna start off sorta more with the theory and the architecture so first we start with V PC with the internet gateway pretty basic we we have to availability zones subnet is specific to an availability zone so we create a subnet in each one we put a VPN instance in each subnet pretty straight forward from there what we do is is let's add a spoke to this so what we're going to do is create a VPN connection from one of these boxes so that VPN connection for the purposes of you know this presentation and there's already too many lines everywhere so the VPN connection when you create a VPN action actually has two VPN tunnels served time I draw a line up here it's actually two so keep that in mind when we started out with VPN tunnel count for the rest of the presentation I'm just gonna sort of ignore that over that VPN connection we're gonna run BGP that's doing our dynamic routing updates and from there we're able to propagate the routes from that we advertise from this VPN instance into different subnets so on each route table you have the option to basically propagate routes from the vgw it's basically a checkbox and so when you do that when you look at the routing table you'll see now we've got a route to the 10.0 / 16 network which is in this case the transit BBC and it says that it was propagated and the route is to the vgw so the good part of this is that no matter what happens to those tunnels that route doesn't change the route still points to the vgw it's highly available so that's a really important important component of why we're using the Virtual Private gateway here so the V GW by default is going to advertise the the V PC side arranged so in this case is the 10.1 / 16 and then the good part here is actually this looks exactly like what you would do on premises so I just gave you the VPN like primer but from the purposes of the Virtual Private Gateway it has no idea that we're doing some sort of funky transit vbc thing right it thinks the transit V PC is a customer gateway it probably thinks it's on-premises or something but effectively we're not doing anything different with a VPN we're using built-in functionality which actually one of the nice things about this is that if this is not your V PC this is maybe the developer or someone else's V PC you're using totally native services and there's nothing in there so it's very easy for them to sort of have this demarcation of yeah whatever's on the other side that vgw I don't care I don't to manage it I don't to do anything so that's that's a nice component so let's add another one so we create a VPN connection to the other instance then let's add another spoke so when we add another spoke we're gonna create two more VPN connections to the virtual private gateway so full mesh basically inside of that we're gonna have a route to the vgw and case you really don't need that 10.0 / 16 they really just need to know how the other spoke so for simplicity we're just going to show that the 10.1 round from there the the VPN instances were going to basically route to each other so in packets come over the vgw they will be sent to the other spoke and you can control what gets advertised from these transit VPC instances so that allows you to do some control by default we're just going to assume for the rest of presentation basically that the Virtual Private Gateway we advertise is everything it receives so this can all be used for the internet so some people want to centralized internet connectivity you can do that what do you do there well we just have the the transit PPC advertising default route and now we can make that work and you know one of the common questions I get is you know why don't you just use VPC peering it seems a lot easier than VPN and it's faster and all that sort of stuff so let's let's find out why not so we're gonna create a V PC parent connection from the spoke v pcs to the transit V PC and the spoke V PC we're gonna create a route basically we'll just gonna try to emulate that last slide and we're going to have a default route to the peering connection so in this scenario it just says if if you wanna go the internet go over the peering connection and they'll figure it out later well that doesn't work so in this case the destination IP address is the Internet when that packet gets routed within that transit V PC it says like hey I need to go to I don't know 54 dot something where do I go well it says will that be the internet this doesn't work that is that is what we would call transitive routing and so effectively if you want to think about sort of a theorem I don't know if this is a law but it's a theorem at least that if what the way to define transitive routing is either the source or the destination for every packet needs to be a network interface in the local V PC so in this case I'm coming in over V PC peering and trying to go out the internet neither one of those is a network interface in the V PC that means no bueno which is Spanish for not routing so yeah you have to worry about that so change of routing us that's one of the reasons why were you doing this so if we go back and we take a look at this with the VPN ones up happening is we've got the VPN connection the next addition is the Internet it goes to the vgw vgw terminates the tunnel on the VPN instance and so it lands on the you know the VPN essence here from there the VPN instance is going to afford it to the Internet in this case the source IP address or the source is the network interface of that VPN instance so now we're winnow again we're now forwarding so that's that sort of why we're using VPN so let's take a look at you know most of time when I talk to networking to do about something I explain it you say very nice and they go how does it break so let's find out in this case we're gonna break one of the VPN tunnels either because something on the ATS side broke for that tunnel or someone missed configure the tunnel something happened on that that one tunnel what is up happening here is there's some protocols there decades old dead peer detection which runs on VPN it's on by default and BGP has their keeper lives so from a type of timer perspective we're gonna understand that and sort of one of the nice things here is that we're also not using any sort of proprietary failover stuff this is decades-old networking protocols and as I discussed earlier the route inside the spoke VPC points to the vgw that VW route will automatically switch over to the other tunnel so you don't have to do anything in the spoke VPC which is really important so let's take another let's break something else let's kill let's kill an instance things always fail right so instance fails what happens so both those tunnels go down and again the the spoke VP sees again they're gonna have timers so the both those timers should detect that failure within 30 seconds as those the default BGP timers and the default dead peer detection timers so that's on by default it's just there and that's going to work so that's that's a really nice availability kind of concept for for AJ I've want to connect on premises to this what do we do there's a few options so the the first sort of more so obvious one is just do a VPN over the Internet we can also say well look man I don't really like the internet but much I want to do Direct Connect I want it to be private ok old Tyrone tunnels over Direct Connect and then we've also got this sort of this wonky thing called detached vgw and you hear this a lot about the transit VPC specifically so let's dig into these options a little bit so the first one is going over the Internet so in this case we've got our same topology we've got some on premises data center stuff and you just create tunnels and you terminate them on a router or firewall or whatever you like on premises so that's pretty straightforward the reasons you want to do this is we want to see on a pretty simple allows you to sort of really control that tunnel because it's terminating on a device you have full control over it also gives you the functionality if you want to do some other some other options for the tunneling so if you want to do GRE like an unencrypted tunnel potentially for higher performance or if you want to use something like the the Cisco dmvpn to do is sort of a multi-point VPN you have options of how that tunnel works but it is manually configured and operated right and it that tunnel also has to terminate at one place or you create more tunnels if you want to terminate more places and that depend upon where the traffic's trying to go that you know it could be a challenge for a Direct Connect I'll walk you through direct Knittle bit the first step is figure out which direct connect location you want to be in from there we have some sort of way of getting to that location so that could be at least fiber that could be a partner network you could already be in that so if you're already in Equinix and we're in Equinix and all you needs a cross connect either way you figure out how to get to that direct connect location through sort of networking you put a device there that's going to do the BGP and VLAN components of Direct Connect you would create a virtual private gateway attached to your V PC and then the last step is creating a private virtual interface or a private if from there then you just create ahnold's so if when we think about this design it's great because if you want to encrypt over Direct Connect this is a great way to do that so you're using a private circuit you're getting predictable latency those types of things you can also again since you have control of those tunnels to use alternative tunnels like giri and dmvpn but again we're still mostly manually touching and operating this thing so let's take that same use case here we've figured out our Direct Connect location we've got a device doing that stuff this time we're gonna create a virtual private gateway we're not going to attach it to anything we just literally do to go to a virtual private gateway create new you've now created attached vgw from there and your Direct Connect configuration you create a private virtual interface to that virtual private gateway so it just looks like a normal again from Direct Connect they have no idea what's attached or not it looks the same now one one note here and this has been a recent addition is direct connect gateway requires you to attach it so this is not compatible with Direct Connect gateway so there so then what we do is now we create VPN connection from our VPN Asus's to the detach PGW why is would you want to do this well what's sort of nice about this is now you can sort of consider on-premises is just another spoke so from a design perspective it's nice and consistent right so it looks the same when we get into the automation components you know it's a lot usually automate this because we're not doing sort of one-offs you know it's automated in the case of our transit VPC another advantage is once the the traffic comes off this the VPN instances on to the virtual private gateway and it goes on to the direct connect network the direct connect after Direct Connect is your it's your it's your network so you can sort of fan that out to multiple locations all that type of stuff so allows you to sort of fan-out out connections and the VPN termination happens closer to us you know and further away from our premises so you can sort of go multiple places but you know if you if you do care about encrypting your Wang connections this is would be unencrypted after it leaves the virtual private gateway so let's let's take into automation a little bit this is sort of the cool part so if you're not familiar with cloud for me it's a service that we have that allows you to do things like infrastructure is code essentially it's a big JSON or Yambol document that describes what your architecture or your configuration should be so in this case we go back to our sort of hub in this case we put Cisco Cloud Services routers or CSRs the CSR is if you know Cisco IOS it's that in a ec2 instance it's available off of marketplace or you can do you can talk to a Cisco people and get your own license we call that bring your own license or BYO l you know one of the reasons why we chose Cisco for this automation solution is that they inside their operating system they have support for virtual routing forwarding or brf's which handles a special VPN case of duplicate addresses which we'll get into that a little bit as well so let's talk about this automation looks like so first off you have your CloudFormation template you can choose a speed and feed run it in the region of your choice from there it's going to create what we showed earlier by default is going to use a 100 dot 64 which is a carrier grade nat sort of special IP space you can change that it's going to create an s3 bucket so that's where we're going to store our VPN configuration it's also going to create an s3 endpoint so that we can access that s3 bucket privately and create the route so we can get there as well we had talked about killing instances before you can now do easy to auto recovery so Auto recovery means if that instance fails we detect that failure at an ec2 level then whenever we detect that we will boot it back up and start again so we added a little additional H a to our previous you know scenario so what does it look like to add a spoke so in this case what we do is we put a tag on the Virtual Private Gateway so when you create the CloudFormation template you can specify what you want that tag to be so in this case the default one is transit v bc you know spoke equals true what happens is there's a lambda a time lambda running every minute looking for that when it sees that it goes oh looks like we got another spoke so it creates a VPN connection for that vgw from there it takes that VPN configuration and puts in an s3 bucket on that s3 bucket we're using key management system Edo's KMS so kms is doing server-side encryption on that bucket to make sure you know good security practices we're encrypting things are important like VPN configurations from there the s3 bucket there's a lambda watching at s3 bucket for changes when it detects the changes it downloads those configurations and uses basically a Python thing to basically send those commands down to both of the routers so there's a security group that allows that lambda function to securely communicate with those instances so in this case the CSRs have now been given the configuration for the VPN and presto magic we've got VPN connections and all we did was put a tag on it so this is really attractive because now people that don't know networking you just tell them one thing like put this tag on it poof magic network connections so that's that's a really nice benefit if we want to remove this book for whatever reason looks pretty similar we just change that tag from true to false so let the time delay no function is also looking for that so it sees that change knows what configuration you have goes to lambda says hey go delete all the configuration and then this the csrs will delete those VPN configuration so just in Reverse if you want to do this to another region so if you got a VP T in another region that you would want to add to this all you have to do is just tag it so the lambda function looks across all the regions and automatically does that so that one we're good we don't have to do anything sort extra work there if you want to this in another account there's a few steps we're gonna have to take so in this scenario the lambda is going to run in the the spoke account and there's three things we have to do one is we have to setup that that vgw polar that lambda function we need to allow a bucket access so it can access the bucket in the other region and it also needs to add allow kms access so the first step here is is doing the polar so the way this works is with there's a separate cloud formation but you can run and the spoke account and that spoke account is essentially sets up the lamda sets up the timing looking for the tag you know those types of things the next one is allowing bucket access so we need to make sure that our s3 bucket allows this thing to function to put things in there so there's a few ways to do that so this is just an example what the example code looks like you basically add another account in your s3 bucket policy optionally when you set up the CloudFormation template the first time the main template you can specify up to one other account that you want to add as a spoke if you do that this happens sort of automatically as well for kms kms needs write access to this bucket so you add another account ID to your key policy and again same thing if if you want to do this that it's set up you can allow one other account and the CloudFormation template and do that otherwise you can go edit this this key policy so we've done those three steps presto magic we've got V pcs running in another account right so let's talk security a little bit so I know this conference I've had a lot of conversations about you know firewalls and inspection and how how do you do that in the transit V PC so let's take a look at putting firewalls there instead of VPN as this is doing routing so you know first the first thing is like why and we asked that a lot you know because just because you can doesn't mean you should it's sort of a you know a lot of the questions I get with customers around they have firewalls and you know the cloud in general so you know my opinion is that there's probably a lot of people who went to a database to get away from their firewalls so you know operational things you're doing with your firewalls are actually important so just because the technology works you also make sure your operations in your automation because you know we're just gonna be back in that same thing where you know you put in a change and now the firewall engineer takes 45 days to make a security change instead of the network guy and so be careful about that sort of thing but the types of customer use cases I'm seeing for or maybe abs is considered like an untrusted data center maybe it's for bursting or whatever reason or they just don't trust ATS yet so they want to put a firewall and inspect everything that usually goes away after time by the way they tend to trust us as they start users meeting us more there's also compliance requirements around intrusion detection intrusion prevention sometimes contractual sometimes security controls there's also security organizations really like like next-gen firewall and application level policy inspection those types of things sometimes that's the requirement as well you know once you start doing this you if you've got 100 V pcs you probably don't want a hundred firewalls so like how do you centralize that and it also helps you get more visibility sometimes when you centralize that too so this that's why this tends to be a pretty common pattern so if you go back to our diagram now we've replaced the VPN it's this is with firewalls and it's mostly the same one of the big differences is that firewalls are stateful that's sort of like their whole purpose so what you want to do is do a s path prepend on the firewall so that one of the firewalls is a more preferred route you can do that with the routers as well if you like you know determinism and knowing which in your routers is active passive both firewalls it's a basically a requirement so let's talk about some issues here too right so there's this issue of duplicate tunnel addresses so like I said that's one of the reasons why we chose to see its are for the transit PPC so if we take a look at this a little bit deeper when we create the VPN connection there's a VPN configuration that gets pushed down to the firewall to create the VPN connection if we look inside this guy we'll see something like this so this is just example Palo Alto configuration so up top and below we've got a 169 dot 254 address it's a slash 30 point-to-point that address is randomly generated by the Virtual Private Gateway so that's good we don't to do any work but let's create the other VPN connection now to the other vgw that vgw doesn't know anything about the other vgw they don't talk they're separate so he also randomly generates a config raishin and in this case we roll the dice and we got unlucky the both of these V GWS gave us the same address so they're both expecting us use one six nine two two fifty 4.45 to ten so now we've got options we could we could just try again we could delete that VPN configuration and just create a new one roll the dice and hope for something better the way the math works house is sort of interesting it's the birthday problem your statistics person so there's three hundred sixty six unique birthdays at more than I think about twenty two people in the room someone's gonna share a birthday so I'm pretty sure some of you guys share birthdays but the same math applies here and basically somewhere mathematically speaking someone were like eight to ten VP sees you have about a 50% chance of getting overlapping address on a new VPN configuration so firewalls typically don't support duplicate addresses on their interfaces so I've got some good news maybe six weeks ago or so we sort of quietly announced this feature to define your own tunnel address on your VPN configuration so in here you can assign both the inside Sider for both tunnels that allows you to define your own address now you do have to do some basic IP management to make sure that you're not giving yourself the same address twice but you do have the control to create your own addresses on the VPN configurations this also applies if you're doing lots of VPNs back on premises so if you have like one big router that has a VPN configuration too and I've I've seen it with customers that have VPNs to hundreds of e pcs if you take that approach because you're one of the people that likes tunnels this also it makes that life easier for on-premises VPN cool so there's another approach here that that we see is okay well why are we using the virtual private gateway can I just put another VPN instance in the spoke VPC and do VPN there and the answer is yes you can do it but it works a little bit differently so in this case we put a VPN instance and each of the spoke fee pcs and what we've done is we've created a route table entry to the network interface of that instance now you could you know run multiple instances and and there's some other high availability tricks around you know moving the network interfaces or moving routes but we're not really gonna get into that but you'll need to figure that out as part of this so you know this is good for you know to get you even more control because now you know you don't wait on the vgw and you have full control that instance and you can do whatever you want you can do all sorts of tunneling you can also do other functions like firewalls so you can put firewalls in the spoke VP C's but now now we're managing things right like ATS likes to have managed services customers like managed services what I've found so now I've got to manage these instances or firewalls in each V PC so you want to think very carefully about how you're going to do that automation is important right but you know one of the things I said before is if this is not your V PC you know if this is developer you've now put an instance in there V PC you know like man I don't want an instance of my V PC that's not mine it's yours and I don't like it so it's you know it's not native it's a little bit more intrusive than that V PC but you know if the way if you figure out the policy you can sort of make that stuff work and one of other things you want to think about is how you gonna get routes into those routing tables so if there's a default route it sort of makes sense but if you're doing some sort of BGP we no longer have that sort of convenient checkbox feature to propagate routes so that makes a little bit more challenging if you have sort of complex routing requirements that need to go to this this instance near V PC so we've got another sort of design here around this so if we expand this out a little bit and maybe we've got multiple availability zones and what we're gonna do here is we're gonna put a firewall or an instance in every availability zone or potentially every subnet depending upon what you want to do and what we do is then do a full mesh VPN configuration for all these guys so why would you do something crazy like this so in you also have the routes pointing to each network interface but it's it's really scalable right like there's no limit Tommy like instances you can have in this like just go crazy there the tunnels like I said tunnels are highly scalable so you can do that if you have a firewall or innocence in each availability zone then you know if you lose one of those then you lose one availability zones hopefully you're running applications that can support that level of failover if not this you know may have some issues for you but if you've got it you know applicational will failover you can lose one or one of these and it should just route around it should be fine from application perspective you know some of the other component you want to think about is ganas centralized management and overhead you know if you're choosing a sort of solution that uses this type of design they typically have that but you do want to sort of keep that in mind the negatives are if you guys pay for firewalls you guys know that it's typically by the box so now we've got lots of boxes so we've increased our licensing costs another one is just sort of like the high management overhead right like I said managed services are nice when we can use those and the route propagations still a problem I do want to get a shout out here - - Barracuda because they actually just changed their licensing model to be bandwidth based so that box based licensing is not there so it's a nice little feature to help out so let's talk about the rest of the ecosystem so I talked about Barracuda there's some other folks you know obviously we took we chose the Cisco CSR when we built this solution out so there's a lot of folks here that do sort of more routing based functionality using the vgw so if they're using our VPN product as the spoke and there's a bunch of people doing firewalls - so this whole group of folks but effectively you know really anyone that's doing both bgp and VPN and they could support that to our virtual private gateway it really could be anybody if there's some other logo that I missed here and there's another sort of group of people that are doing this instance based sort of transit VPC and and this is basically anyone can just do tunnels because just instances with tunnels will join each other right so if we then we can actually classify these further by the law of automation so I'm going to start with the folks that what I would do consider like continuous automation so this is sort of the transit VPC where you know you're getting tags and you're seeing it and things just sort of happen automatically so these folks in orange are sort of on that there's these other folks in blue that you know they have a confirmation template to spin up sort of your first transit vbc but then usually you have to do something per spoke manually or some clicking is involved and then you know all the folks in green is basically sort of they may have some tools or it may be sort of do-it-yourself but like I said anyway they can do like BGP and and tunneling and VPN can build a transit VPC but then it's really about the automation component cool so I got bored I made the cool animation but let's let's talk about the cost scale that type of stuff let's go that's another question I get a lot of so let's dig in a little bit there so this is our transit vbc let's go down so per spoke there's a charge per VPN connection so there's two VPN connections in this this design so double the hourly cost per spoke there's also the egress charge to the transit vbc there is also aggress at the transit busey itself so traffic that's either going back on premises over VPN or Direct Connect or out to the spokes coming maybe from there so there's some transit charges there inside the transit vbc itself there's obviously the ec2 charges of those two instances and any may be licensing costs or you can also use open source but that's there as well if we take a look at some of the performance I work really closely with a Cisco team so they helped give me some of these numbers and testing they've done on their platform so this is basically what you'd expect from a performance perspective mostly for reference I don't expect really anyone to memorize this but do you want to call out one thing is that the packet sizes for this are you know basically perfect packets you know 1,500 1,400 byte packets so if you're not sitting those that you may not get these numbers so let's dig in a little bit there so we got our familiar nice strands of DBC again basically from a security perspective we really recommend using security groups inside the V PC that's going to save bandwidth costs and make your life a little simpler if you can each spoke it's gonna have to VPN to VPN configurations and so each VPN tunnel itself can do about one and a quarter gigs roughly you know you might get more you might get less depending on packet size if you have more than 100 V pcs you're gonna run into routing table limitations because by default each spoke V PC advertises its site or range and it gets really tight out to everyone else so at 100 V pcs you'll need to change that routing so you can customize that or you could just create another pod of transit bbc's each of these CSRs or VPN instances and this is basically true across everyone on that slide I had with the pretty colors and stuff everyone's somewhere between about one to three gigabits per second depending upon packet size so sort of think about that as you think scaling out as well for high-performance applications that maybe that bandwidth limit seems low you can still use UPC peering right you can still do that for direct connectivity you know maybe on a one-off perspective so that's still pretty easy to do it makes it more manageable and you know sort of on an EDA basis alright but we're talking we're still thinking like hundreds and thousands of e pcs what we do now because some of these limits don't seem that high well we just make more so you have one transit EVC great another one you know when you do this you probably want and this could be multi region this could be in the same region you know one thing you want to think about also here is you probably use different tags so by default your movies the same tag so you really want to have you want have different tags for different transit PBC's so think about that and then to get east-west connectivity between these things you know just create some tunnels between between them and that's sort of how that you can scale that out cool so that's what I've got on transit BBC's you know obviously it took me a while to sort of get there it's pretty complex but obviously there's lot of value there so let's talk about Direct Connect because a different sort of way to think about this so here what we've got is on-premises on the line we've got our sort of VP sees on the right we've chosen our Direct Connect location we've got a customer router as an ADA best router so this is pretty simple right so basically for each PPC you create a private virtual interface for Biff that's a VLAN and a BGP peering session you can get up to 50 virtual interfaces on a single port for that's for one in 10 gig direct connects for just the purposes of good practice we're gonna make this highly available and you know we're gonna create two more virtual interfaces to each Virtual Private gateway and there's a there's actually a lot more sessions like Steve Seymour has a deep dive on Direct Connect and VPN if you're really curious about how this works and how you set this up there's there's a lot more information on that but we're gonna focus on the scale here so for that physical port get you to 50 virtual interfaces which translates to 50 v pcs there's also that physical port is 50 so we also have a feature called link aggregation or lag so you can link aggregate up to four links per Direct Connect so that now multiplies by the 50 so that's now at 200 so that's a little trick to get you higher than that so you you may want to think about that as in terms of your bandwidth do you prefer to one gig ports that both have 50 or 110 gig board that has 50 you know you can do some math there and figure out what your requirements are and then we've got this new thing called you recognize gateway which actually makes this better so the ones starting point that at this point we have to talk about is it needs to be in the same account so that is what you need to think about but from here what we do is we created Direct Connect gateway we create one virtual interface from each of our routers and then we attach those vgw to the direct connect gateway so this scales up pretty high you right now when it launched to think it was 30 but now it's 10 so you can you can launch up to 10 virtual private gateways per Direct Connect gateway so that's that's nice and scalable if your accounts and the math all works out the right way we have 50 virtual interfaces so I can go to 50 Direct Connect gateways those Direct Connect gateways can each scale out to 10 V pcs so that gets you to 500 on a single physical port so that's a it's a big increase in we're pretty happy about that so direct next gateway is pretty exciting it also works multiple regions so multi-region connectivity you create a couple V B C's in another region you just do the same thing you attach the Virtual Private Gateway with a little fast so it looks basically the same so this is really nice so now you can connect if you have Direct Connect in one location you can now connect privately all over the world pretty awesome so let's talk about shared services so shared services vb c s are pretty common i see these these typically have workloads like maybe a Microsoft ad server or maybe you've got a DevOps toolset that you really like or maybe you've got a logging and monitoring server security services those kinds of things and what we typically see here is you know BBC peering so you create all those things in one V PC you do vb c peering to all of your application or spoke TB sees it mostly works you know there are a couple challenges with it so in this case we've got we're back to sort of our 5e PCs and we've got our shared services there's a couple things that we can approve here so first off is you're pretty much allowing full V PC configuration between the two so it's a pretty blunt instrument in terms of routing you can get more specific with your route tables and those kinds of things but you're basically you're basically allowing both bbc's to talk to each other if you just want to talk to one service it's a little bit inefficient there as well do many developers in the room okay okay I won't say anything mean but if our developers I don't know just accepted the defaults on all their V pcs and we have duplicate addresses we actually can't peer these things anymore so that that can become a challenge whenever you want create some sort of shared resource unless you really really like NAT you know you can double mat and all that sort of stuff yeah I've heard a little snicker yeah thanks I think people have tried that before and then also if we're thinking scale we can only really go up to 125 for this so after 125 V pcs either we need to duplicate this shared services V PC or we need to start using Internet connectivity or something like that so we launched this week a feature called private link so we really said a couple weeks ago and that was for a two best services so for like the ec2 API our load balancing API etc but what's interesting now is that for customers and partners you can create your own endpoint in your own VB C's so the way this works is up top we've got sort of in this case what we're gonna call the shared services VP see we've got some sort of resource in this case let's just call it an API because it's sort of simple to visualize and down in the bottom we've got our spoke so what we do is we put a network load balancer in front of these these API servers that's going to do our load balancing we associate that network load balancer with an endpoint service so in this network load balancer also gets a single IP address per availability zone so that's sort of nice if you like the virtual IP concept from here we create the endpoint and those IP addresses just sort of magically show up and the other VPC and so and these use addresses from that that spoke VP sees cider range so in this case we're 172 16 up top we're 10.1 on the bottom the IP addresses are in the 10.1 space they just pull sort of random address there but it's a local IP address so you know what one thing and the way this sort of works is that you know it's unidirectional access so the spoke BBC is allowed to request the client VPC and obviously you know responses as part of the TCP connection come back but things up top can't initiate connections down to the bottom so that's good from a security services because if someone gets control of my API server they can't do anything to all my spokes which is different than BBC peering so that's nice from a security perspective one of the other advantages here is if we've got sort of another spoke maybe our kind developers created another VPC and they also created 10.1 / 16 this works so you don't have to worry about it because it's unidirectional sort of functionality this just works so you know you can do overlapping addresses with this service so that's nice and you can see in this case I randomly chose some other addresses for this 10.1 just to show that you know there's some randomness there but the way this actually works is inside that VPC you'll get basically a DNS name for this service and in each V PC that DNS name is going to resolve to those IP addresses in s spoke V PC so I'm going to API example.com it's going to resolve to ten dot one dot two or whatever is something in my local V PC so it's split horizon DNS so that's the way that functionality actually works so this also scales out to thousands of V pcs so again this is the same thing we're using for our services so the ec2 API as you can imagine helps thousands of customers do that and you've now got the same scalability when you build services so there's also a design for this using the transit V PC so I see this fairly often and just sort of like we made our on-premises Network look like a spoke we can also make the shared services just a spoke so again it just sort of works you'll have to worry too much just one note I do see some people wanting to put the shared services UPC inside the transit vbc I tend not to prefer that simply because it changes the the access model you now have to worry about routing to local network interfaces and you're now back in the game of shifting routes and shifting network interfaces those kinds of things so this is really the model I I prefer but like I said man I'm just just my opinion man so let's compare this compared to some of these options so probably link is really sort of the place you should be starting so if you can provide services through private link it's a better way to do this but it is unidirectional does use our network load balancers so those are sort of dependencies that you need to think about but you know if you got overlapping addresses it's really handy to do that BBC peering is sort of you know it's still there it's still solid it still works but you know think about the scale there and think about that security and for the transit vbc you know it's still fairly complicated so you know you're paying a lot it's operationally more complex than both of these but you know if you really like to transfer BBC it's an easy way to continue doing shared services so just fits in that model pretty cleanly all right let's talk about mental regions right so this one actually got pretty simple this is my favorite slide because it was really simple you can you can peer multiple regions now right that's great and it just just sort of works I mean it's encrypted so we do some some pretty cool encryption things on the back end to make the the VPC peering work so it's encrypted runs across our backbone so it just again it just works if we compare some of the other inner region options that we talked about you know direct connect gateway is great you know it was just for direct connect so if you're especially if you're in the same account it makes sense to go do that for global connectivity for in a region peering yeah we still may have that challenge around the fact that it's it's one-to-one but you know with all the customer conversations I've had this week about it it's usually for like replication or a very specific use case so you may not have the requirement to have like many to one v pc p-- earrings but it's really depend upon your your architecture and you know transit vbc a lot of people are using the transit V PC for this today still works if you want to you can also do it over cross region peering so you could create a cross region peering and then do VPN over that if you really want to maybe if you want control of your keys or something like that but you know we've got a pretty secure key story so I'm really I really hope people are using in a region peering and Direct Connect gateway here so this is a slide I I really like so this is so this is sort of how I usually walk down the customer conversation in terms of scale so you know between like 1 and 5 V pcs like I said most of stuff at the beginning you know really covered what you need to know it will it works right it's pretty simple it's pretty easy to understand it's really when you start getting over that it's you know that's when you so we're not talking about automation that's when you start what I'm talking about again it like maybe 10 to 15 v pcs that's where we run into that duplicate address problem we start running some complexities there at 50 that's where we tend to start hitting some of our limits so that's the default route and peering limit so you know you can get that raised up to 100 as well 50 50 50 VCS is also where we're gonna start thinking about Direct Connect gateway so that we can you know start fanning out our virtual interfaces a little bit after 50 then we get to 100 or so at a hundred we have to start thinking about that route limitation so that's the amount of both either static or dynamic routes that you can have in a V PC like I said the transit you see automatically advertises it to everyone so at 100 pcs we need to go customize that either we might want to advertise like the entire RFC 1918 range or we might want to advertise a default route out of the transit BBC so that you know we don't overrun that route limit 125 that's where BBC peering that's the maximum for any given V PC so after that we may have to start either using private link or start using or duplicating services or using the internet and at 200 that's where you know maybe we did a four by lag so that's sort of our virtual interface limit at that point so it said Nick you told me you were to talk about thousands of V pcs okay let's do it the answer is straightforward but not use the internet right the internet is inherently scalable and there's there's a couple tips here you know so because it's really about security once you start going over the internet there's a couple hints here so secure protocols so use SSH you know use use TLS use secure protocols make sure the authentication is strong as well so don't use passwords please use the access use the the certificates and keys as part of this so that we don't have leaked passwords a bastion host or a very common sort of approach here essentially for what that is is basically an instance that is open to the Internet but it's sort of your jump post so you can sort of control that a little more tightly and allow that access to your resources and that Bastion host is just on the internet you can do that private link is also a really good option here because of because again because of the scale of thousands so that's one of the reasons why we really like that feature cool so let's let's talk a couple little tips and tricks it's a little bit of a little detail here so I guess I've been Network guy for a while you know I spent a lot of time with customers historically that we're buying hardware and they go like okay well do I need the 3 or the 6 for the 9 or the 12 version of this thing like what's my network bandwidth going to be and you know 7 years because that's how long our hardware lasts so I need to figure out what our network bandwidth it's gonna be you know I'm gonna spend months and months looking at this and price comparison and you have to do that anymore like stuff changes like you can you can we bill you by the second now so you can just test it out try it see if it doesn't work you know we also released like a bunch of features here - you know this week so things change pretty often so even just the recommendations I give for this type of network architecture they seem to be changing every six months right we're releasing new features partners releasing new features scales changing and so like trying to create like the perfect custom architecture that's gonna get you through the cloud for the next seven years is waste of time so really try to focus on you know probably the next year year and a half or so depending upon your requirements and don't be afraid to sort of mix and match these things - right like these are just recommendations so like segmenting out and also combining these things right so you could say well this for this environment may I use trans people see this environment maybe I use inter region peer in a private link like that's okay like you don't have to choose one of these things and testing texting is testing super important and then you need to mix and match these things up - cool so this is mostly for reference I'm gonna skip that this is how you can customize some of these things there's some lambda functions and then you can use tags and some use cases you can order your customization - so this isn't day let's so we didn't really talk to me a customer use case so Vicente is gonna talk about hit what they're doing and how they customize it oh thank you very much Nick hi everyone the same to the Luke I work for it a network engineering team at Zendesk where most of my job is make sure that packets move fast from A to B without any laws and also in a cost-effective way so for those of you how any people knows end a seer well that's great so for those of you who don't know I'm inviting you to check our website we are a company that builds software for customer for better customer relationship so check out our website and sign up for a trial so coming back in 2015 this is how our internal backbone looked like we are we were a traditional network internet shop at that time we've run data centers in United States and Europe and we relied on multiple VPLS providers to interconnect all of them and we also deploy BGP so we could ensure high availability between those tunnels so early in 2016 we deployed our first V PC in Amazon and the way that we choose to interconnect this V PC down to our existing network was by using the manage VPN over Internet and public peering and the main reason of that is that Zendesk pretty much wasn't asking you know not loves each other so we already rely on Internet to serve our product to our customers so why not also make sure that our machine to machine Talk can also use the Internet and to ensure that we were high available we deployed the best practice that AWS folks already covered in some of our older talk so we've basically run - Fargo spur data center and active standby mode this end up with 16 IPSec tunnels into this model but only four are active and we have to play with BGP prepend and local preference to ensure that the BGP is matching the network topology so this thing worked very well and later on that year we said well let's do an other VPC let's deploy our application closer to our customers somewhere else in the globe so things start getting a little bit more complex here and most because when we create a new V PC with this model it requires configuration changes in all the existing environments right and this is a big risk and it also we need to ensure that those new V pcs talk to each other and around that time there was no V PC peer in the cross region so we had to deploy ec2 VPN instances and now you can just replace these by V PC period but you will still net heap need to interconnect this down to the rest of your network so again things started working very well we are a fast-growing company so early this year this is how we start looking like so we had four V pcs all of those tunnels and things started getting very complicated especially when we were creating new environments right so make mention that sometimes you have like 45 days delay on setting up new V pcs we never got that far but we still have a little bit a lot of trouble doing that and it was time for designing something and thinking of something else so our proposed solution here was to build a location agnostic Network and also make this network simple and cost-effective for older people to join this so this network should provide ipv4 and ipv6 with encryption right in full mesh topology it needs to support on-premises data centers and multi cloud providers scalable for growth we wanted to describe our whole network stack as code so we could fully automate bootstrap in our rescaling and eventually self-healing in case something goes wrong and we also run kubernetes in production at Zendesk and this is also before eks okay so we name this project Medusa and the reason is that all of the tunnels that we were handling is it reminds the curly hair of Medusa so how our network looks like today so we deploy what make presented is shared services VPC where we have something like LDAP master or service discovery things and we deploy cisco co-stars as dmvpn hubs so if you're not familiar what dmvpn is it's basically a protocol suite to include an IPSec JRE in a jarppi and internal BGP and I'll show you how this thing works so let's say we want to connect our datacenter to this model so our data center would apply to routers running iOS XE and we configure them in a vanilla Cisco - Cisco dmvpn and the configuration at the hubs one minute okay we're short on time so I'll try again real fast here so the configuration at hubs does never change it's everything and let's say that we're now connecting a V PC this is the secret sauce of our solution here so we deploy what we call the Medusa routers the Medusa router is a Linux box and we've deployed the easing to an Audi scaling group and we've set up some open source software that represent VPN stack and we also have some custom Python code to propagate those BGP routes that we learn from the network to AWS routing table some of the motivations that we have it's mostly because this is very portable so we can make this work whatever it runs Linux and let's say that this V PC needs to talk to my data center now the way it works is spoke a sends a packet to spoke B through the hub the hub will send a redirect message back to spoke a and say hey you should better build a direct tunnel to spoke B so further communication will take the direct path and let's say we're now one to scale NV pcs using the dmvpn model this is how it looks initially when they come online but as the traffic converges we now build those dynamic yellow tunnels and as you can see we have multiple tunnels from one V PC to the order so they can also load balancing across it so the way that we deployed is in Zendesk we are a big fan of hashing for terraform so everything is automated in Hashi Corp we also leverage Packer for building a custom ami where we have all of our open source daemons configured in the Linux box and to build a new V PC we require six parameters so you can just input those six parameters hit terraform apply enter and wait like five minutes your V PC will be up and running all connect to the rest so the deployment state is around this is we're in progress of moving from legacy to the new one some of the performance tasks that we run we could increase almost ten times the true put compared to the legacy model using manage V PC VP M sorry and if you're doing this at home there are some challenges here you will need an iPad or a lambda solution to give an ID turn out an x IP if you're reaching the route table limit of a hundred you will want to advertise all the RFC 1918 routes and stay tuned at early next year because we intend to open source this tack and also write a blog article in the battery detail so thank you very much for your time okay all right well we're just gonna skip through the rest of the slides here went a little bit over on time so yeah thank everybody we'll take questions up here up front and thank you for coming

Channel: Amazon Web Services

Views: 42,458

Keywords: AWS re:Invent 2017, Amazon, Networking, NET404, Connect, VPC, Security

Id: KGKrVO9xlqI

Length: 67min 17sec (4037 seconds)

Published: Fri Dec 01 2017