How Devices Connect to the Fabric: Understanding Cisco ACI Domains

all right my name is Carly Stoughton I am a technical marketing engineer with the NCMA business unit which is responsible for ACI Nexus 9k and the Nexus 3000 as well and we I will be talking about today a little bit about how devices connect to the fabric how are we actually connecting what we have in our network today all of our different hypervisors bare-metal servers routers everything we have out there today so we're going to look at the different domains which is how we define these different devices that we have on the network that's one slide no more rest is whiteboard all right so what we're going to start with I have a my spine leaf topology I have my controllers again which is a cluster of three three controllers all devices are connected to the leaves leaves only connect to spines leaves don't connect to each other spines don't connect to each other so I have my standard spine leaf topology predictable latency every device three hops away from one another across the fabric first thing we're going to look at is VMware integration what we call a VM M domain or a virtual virtual machine manager so one way that we can get devices into different endpoint groups is by integrating into our VMware servers this is actually going to give us a lot of visibility into what's going on inside of the VMware servers it's optional you don't have to do this with your VMware servers but what we have here is I've got an ESX AI server that's running say a web VM and I've got a database VM on there so we'll say in my controller I've already defined an application profile so maybe my application profile I've created a web EPG and I've created a database EPG again that was all from the controller so the cool thing that we can do is we can actually from the controller we can push a vmware distributed switch into these vmware servers and by pushing that switch in each port group can act as an endpoint group so instead of do the finding EP g's by things like VLANs or VX landtag or physical port I can do it based on virtual port so the way this works is I will from my APEC set up a relationship between V Center so I'll sent to the IP address credentials of vCenter server used for my a VMware management and this creates what's called the vm m domain a virtual machine manager domain and i'll have one of these per data center so in vmware we've got v center and underneath that we have data centers right vmware construct the vmware admins use to provide isolation between servers and virtual machines between the 8th our controller and vCenter server it's one vm m domain per data center so i if i wanted to have integration i would create two separate vm m domains one for the east data center one for the west data center so that allows you to deploy this in pilot environment while the production VMware environment is untouched right because it's on a per data center basis yep well one Cisco vm m domain per one VMware data center the M worth being the data center object Cisco being the our VM M being the Cisco object so with this integration once I have set up this relationship under my endpoint groups what I can do is I essentially say I add that vm m domain let's save server belongs to datacenter East so maybe I've called this East I add that as one of the criteria for that end point group that's one way that I'm going to sort things into that end point group I could do the same thing for for my web once I do that a minor detail I think you skipped a step at least for me so when you connect a pic with the vCenter server and you have to push the distributed switch into ESX servers is that your AVS switch or is that something else can be either so you could push you actually choose when you're doing the integration whether you want it to be the ABS or if you want it to be a VMware distributed switch also you also work with the standard distress which Louis distributed switch not not standard yeah yeah yeah yeah you know the difference yeah but the yeah the virtual distributed switch mm-hmm that's nice but yeah you can choose it it doesn't have to be a vs it could be regular distributed switch so what that's going to do for us once I tie so I've created the CPGs I tie this what's called a domain in here the VM admin then needs to connect the NIC to the distributed switch so this could be the Avs or it could be what I like to call the a pic created VMware distributed switch what is a V switch that gets pushed in there now it's up to the VMware admins to go ahead and connect an uplink right that's their job but what they're going to see is they're going to see Newport groups pop up that line up to these EP G's so they're going to see one called webs port group called web and they're going to see a port group called database and then it's up to them to go ahead and connect the v-neck to those porkers that's still the the VMware admins responsibilities the port groups are created from a peek out yes the APEC pushes the porkers absolutely so we get that level of control where we have the network folks responsible for the network things VMware guys all they're doing is connecting VM support groups so it gives us that nice line in the sand of who's responsible for what so I'm guessing that this board groups are read-only in the center no and that is because of the VMware API so we're actually pushing in the distributed switch if it's the stand if it's the VMware version of the distributed switch that's going in through their API so that's how they create it so we can't touch it ABS different story that's more like the nexus 1000v where we have control over that so in a so I've got a UCS environment that I'm rolling into a CI so I've got multiple Cisco options now on how I address VMs right you can do VM FX through the through UCS you've got the ability to use the Avs to associate those VMs directly with a CI what's this what's the best design solution there so it's it's going to append on your existing environment and on the application okay right if you're doing VFX then you're probably not going to be doing this integration okay because you're trying that directly to those virtual NICs so okay hey thank you but absolutely I haven't drawn UCS up here but of course UCS could be sitting out here connected to the leaf also and notice I also have switches in between it doesn't have to be directly connected to my leaves I can have all my other switches out here oh yeah but vmm integration and then anytime say my VMware admin spins up ten new web servers once you connects those to that web port group they're going to show up in the controller and I'm going to see that and the policy is going to be enforced so I don't have to go make any changes so very elegant way to do the endpoint groups so how do I identify those VMs being in those four groups when the traffic comes in to the leaf do you enforce different VLANs for different port groups or do you do that by a based MAC address of the VM the port groups do have VLANs one thing you do when you set up any domain as you create a VLAN pool okay so you'll have a VLAN pool that also gets pushed out to the server and each port group will have a different VLAN that's assigned to it okay questions on the EMM domain number one pretty cool way to to do and point groups it's a nice nice way to set that up if we have VMware servers it's also coming for Microsoft hyper-v integration right now VMware so how else let's say I've got other servers out there right we're not 100% virtualized we're not 100% VMware that's just not the case these days so option number two and three are similar let's start with the second one external physical maybe I've got some bare metal database servers right some applications simply aren't designed to be virtualized and actually run better on bare metal so I've got these bare-metal database servers out here now there's still database servers so ideally I want to put them in the same end point group right why would I create a different EPG the whole point of these end point groups is that I lumped together if things that I want to have similar policy similar treatment so the nice thing that I can do is I can create a physical domain and I can tie it to this database EPG and essentially what I'm saying is anything that comes in on this leaf will say this is leaf 101 anything that comes in on leaf 101 on port 1/2 let's say these guys are in VLAN 10 anything that comes in they're tagged with VLAN 10 put it in the database EPG and that could be completely separate from this right there both database servers but I could actually put them into the same database group or hypervisor agnostic I could be bare-metal servers could be hyper-v could be Microsoft or Pham we're in the same clear.the UPG is wholly independent from like the IP subnet right right so you could have two different devices in two different IP subnets in same endpoint group yeah and likewise you can have devices in the same IP subnet indistinct endpoint groups that would need policy to be able to reach each other right right X it's all based on contracts yeah we're not looking at and we're trying to again we're changing the way we're thinking about networking and we're kind of trying to decouple the way that we just think about VLANs and subnets is how we forward traffic now it's about policy in those contracts I have a related but good it's a related question so one of the things that's been our concern is as we start to craft our our policy really the the number of EPG s and contracts between them and how granular you get is a significant that there's there's a sliding scale there you can get extremely granular you can be very broad right and we're having a hard time figuring out how to define that policy properly in that in that sliding scale of granularity versus you know broad so have you guys yet Debbie do you have any good use cases yet I haven't seen any case that is just yet about how people are crafting their policies to find the best mix of that is there has there been anything done to really to focus on that aspect of it to keep that because I mean the policy explosion could be immense and it could add to like it flex it like firewalls and things that we've seen in the past right so we we are working on collateral to show common use cases Cisco IT was you know was one of our pre FCS one of our first customers and we're working on DVDs and white papers to show real-world examples of how that works yes so keep a look at is that that yes and we have a couple books that were recently published one on troubleshooting and one on kind of a CI overview both of those are available today and I believe those also have some examples and best practices so that would be the best place to start with that I may need to see what the newest material you published I probably haven't looked at the books so I'll take a look at those Thanks so that is another way that we can sort things into these buckets that we call in point groups nice way with the physical domain again database server database server just because they're running on two different types of platforms I want to treat them the same so I want to put them in the same EPG so I could do that for my bare metal and I could have other devices connected here too all right maybe I've got some stuff in VLAN 20 I could set that up on the same port alright I haven't just blown that port all right I can say anything also that comes in on that same port that's tagged with VLAN 20 put that into a different oh I totally agree with what you said before that you know let's stop thinking about veal and start thinking about or something that's starting about the contracts and so on but effectively the IP traffic still has to go between IP address a and B and if they happen to be in a different silent according to the subnet masks someone has to do that so I guess that's what you do in the ACI with the anycast gateway right right so the fabric is what will actually do the forwarding but yes we do have the anycast gateway so it's kind of like having an H s RP group on all of the leaves accept all of the default gateways are active it's not H SRP it's in any cast gateway but that way if I have a VM move from one leaf to another it's not going to have to go hairpin the traffic to hit its default gateway absolutely which also means that you sort of set up V RFS based on the EPG send contracts right yes yes we do have the concept we call them private networks but essentially it's a vrf that's how we isolate traffic and how we isolate tenants and make sure that their separation inside of the fabric yep IP subnets really become kind of secondary yeah exactly how do you handle broadcasts within a subnet then right so if a host doesn't also you know an all hosts broadcast and subnet does that only go to members of that endpoint group does it go to like how does that get so ideally we don't want to flood on the fabric so for something like an ARP request let's say we've got a server that has an ARP request since we know where everything is in the fabric if the spines know where everything is there's no reason for us to flood that across the fabric so when that ARP request comes in we're actually going to turn that into a unicast across the fabric and then when it comes out it'll go where it needs to go okay so we want to eliminate flooding on the fabric about for all my NetBIOS traffic Burnett BIOS okay what not fo but you can you can change if you have applications that do require flooding that's something that you can change but then would that flood within the EPG or with that flood subnet wide so that floods within that would be within what's called a bridge domain that's one of the objects that we have which is essentially a collection of subnets that's what creates default gateways and the bridge domains are where you define the flooding behavior let me to show up on Twitter but now we can go back to slash sixteens and all the data center but I probably not actually good idea well yeah but I mean sure you could go for it there we go Cisco validated design yeah great I said it alright so that is the physical domain external layer - similar concept we're saying anything that comes into let's say this is leaf 102 anything that comes into Port 1/7 that's tagged with say I've got my maybe I've got some minecraft servers down here really important really important application that's my innocence so I've got some minecraft servers so anything that comes in here tagged with VLAN 20 put it in my minecraft EPG so I would have to create a minecraft EPG the difference is with the physical I don't have to create another EPG alright I could just add this physical domain to an existing one with external l2 this actually creates a new minecraft external l2 EPG so you say it creates or we create or is that because I'm probably the only one I'm struggling to understand the difference between the two and it's so on ones on the the physical domain we opted to put that stuff in a particularly PG right and then on this it sounds like we're opting to just create a new EPG right I'm not seen the difference am i - no and I agree with you they are very similar and it's part of it as a design choice and the way that we recommend using these is the physical is usually for things like bare metal servers external l2 is usually for say connecting to my campus switches or the rest of my network where I'm going to have a lot of stuff coming in here so is the distinction that we're not mixing and matching with vmm domains as well it's that easier some of your minecraft example here that's just going to be used that for that we're not also putting in stuff from VMware and into the same EPG it you could well actually with this one no this would be a separate domain so this would be its own its own separate entity you would still need to create a contract if you wanted your users to play Minecraft you'd have to create a contract between this and your user EPG but yes this this is its own entity that you'd set up and again typically this is for connecting to your other existing data center switches your 7k whatever you might have this is usually what we do for bare metal servers so would extre would you have multiple external EPG s on that port for different VLANs or ok yeah most likely you would because if is say going to our data center switch I probably have a lot of traffic a lot of different VLANs that I'm trunking know I really can't see the difference between the two okay great so that's another way that we can sort things into ep geez last one is we're probably going to have some layer 3 connectivity connect to an external maybe a way on or internet router so the way that we can do that is what's called an external layer 3 domain so we will actually create a hearing relationship between our router and our leaf and right now the protocols that you can do that with between these guys are oh s PF v - static and ibgp so I've heard that AIG RP has been opened up now have you considered doing that it's cut yeah that's coming yeah and it's surprising cuz that's the Cisco protocol right it's all open open mostly unclear yeah what's not open about it stuff areas are not in there you get some little nuggets for yourself just saying no transit as transit is coming - yeah now now that you wrote OSPF v2 there I have to ask the question does this support ipv6 that's also coming ok it's on the roadmap okay yes that question in good asthma what a plug-in can see ask that question I'm gonna ask my favorite question which is we would do is is I believe ISS between here ah where we're all spiotti was supported cool yeah NOS PF v 3 will be supported as well for ipv6 section so that's coming so that's what we can do to peer of course you still need to configure this router as you would today the apex not going to configure that are you but you can choose one of those protocols to run between the sleeve and ideally this is dual home try drew one router to keep it simple so we don't take up too much whiteboard space but ideally that would be dual home - absolutely so from the external routers perspective is it seeing that fabric as one device or is you've got multiple pairings to it what's going on there so it's going to be just with peering neighbor relationship with the guy it's connected to and actually what's happening in the fabric we run our own set of protocols right tested for our topology the spine leaf that you don't tune or mess with so the fabric is running things like the X LAN is is and then if you're doing external layer 3 connectivity what happens is you can run whatever you want here but anything that's coming into the fabric gets redistributed and we use MP BGP it's all abstracted away from me you know yes from from that external perspective I'm just a figure appearing with one device really yes yeah we'll just see this leaf which we call a border leaf sometimes as a neighbor from that guy uh now can I destroy this beautiful picture little bit I have the connection from the router to the other leaf so I'm your home into a CI fabric will that be one or two routing adjacencies so if we're going to two different devices that's going to be two different adjacencies and i guess you also don't recommend running for channel over those two links you could so you can because you can do either physical layer 3 interface sub interface or SPI routing so you could yeah you could make it a port Channel that would be fine but M lag between the leaves so yeah you do M lag between the leaves right do what might be Jesse link aggregation so one portrayal yourself yeah oh you cannot host can either be two layer three fabric fundamentally right so host can either be dual earth reattached like routers or they can be lowered to attached with a V PC line yeah absolutely V PC we can still do V PC again I've just drawn one connection for simplicity sake here but let me go ahead and ruin my picture that I can still do that was Kim not me I'm still do a virtual port channels so from the routing perspective it looks like your fabric is the MPLS VPN cloud and the router is the CEO outer essentially yeah we can kind of think of this as as one big router thing that is what the goals again yeah and anything so that one of the at least one of the spines will act as a route reflector so we don't have to have a full mesh bgp topology so MP BGP is what the fabric uses and then of course anything coming out will redistribute into whatever protocol we're running on that link mmm I have to open this kind of work imagine you have a different router sitting somewhere else connected to the same fabric and you're using OSPF from that router to the fabric to this router will the internal routes on intra area routes on that part appear as intra area or inter area routes on this part so the fabric was not designed to be a transit network but there's enough requests for that that that's coming right now any routes any subnets that you've marked as public are the ones that get advertised out but right now if I had another router here any routes that I've learned from this guy we're not going to advertise out to that other router okay we don't want to be a trance in that one yeah it's coming hmm there's enough I mean lots of people do that and that's a common request but right now we're only a ver type advertising the fabric public subnets yep all right so that is how we set up the domain so doesn't matter if it's VMware we can do this cool integration it could be bare metal servers Minecraft servers hyper-v and whatever we have out there that's the key it doesn't doesn't matter we have a way to connect it to the fabric we have a way to put it into endpoint groups

Channel: Tech Field Day

Views: 361,393

Rating: 4.8670888 out of 5

Keywords: Tech Field Day, Networking Field Day, Networking Field Day 9, NFD9, Cisco, ACI, Policy, SDN, Software Defined Networking

Id: _iQvoC9zQ_A

Length: 30min 22sec (1822 seconds)

Published: Sun Feb 15 2015