Deploying ACI Multisite from Scratch

hi my name is robert burns I'm a technical leader with the Cisco Technical Services team that works primarily with a CI that today we're going to showcase is deploying a CI multi-site from scratch so what we're going to kind of cover is first we'll go over an overview of the topology of the to physical fabrics we'll be working with today we'll talk about the IP n how everything's connected so we have a good understanding of what we're going to be working with next we'll go into the actual deployment deploying the OVA of the MSC VM which is our controller and we'll go through the installation process do the configuration and get everything set up next we'll cover the inter pod network configuration and this is again outside of the multi-site controllers realm it's not going to manage that for us that's something we have to configure for us so we'll walk through that for today we'll be using a 9200 nexus as the IP end device next we'll go into a few tasks that we have to do on each of the various fabrics or sites that we want to add to the MSC to be managed so we'll go through the steps that we have to do for that next we'll go into the actual adding sites or fabrics into the MSC we'll get the matted to show you what that looks like and as well as we'll start creating some of the policies on the multi-site controller and deploying those in a stretched manner across the various sites now before we get started there's a few requirements that you need in order to do this firstly is going to be you need the apec 3.0 software and of course the switch software to go along with that so it needs to be 301 or later on the MSC side you're going to need version 101 or later you're going to need a switch for the IP end device and this can be any device that's layer 3 that supports OSPF and jumbo frames was really the only dependencies that we have and the big requirement is going to be on the spine hardware so the spines that you're connecting to the IP n do have to have certain hardware capabilities and that includes having EE or later fabric modules and for the line cards themselves they have to be IX or later line cards so e^x or F line cards are if you do have spines that do not have this hardware they can still be used in your fabric they just can't be your inter pod connections so only the spines that connect inter pod network have these dependencies and for early switches they can be any model does not matter at all okay let's go ahead and get started okay so let's take a look at our topologies here so I've got two sites and I'm going to call them fab three and fab six using my to a pick sites on the left side here fabric three I'll just kind of take you through it so I do have an IPM device the very top there I've got a single spine switch now these spines which is for multi-site support any spawn switches that are connecting to your IP n do have to be e^x or later so the line cards have to be X or F X the fabric modules have to be FM e or later so there are some requirements area which we do have very well documented now if you have spawn switches that don't have the e X level of hardware you can still use those for inter site communication but in terms of which you can connect to the IP n they do have to be e X or later now on my ly switches there's no restrictions they can be any level any generation of hardware so I just have a simple pair of leaf switches my APEC is a single APEC cluster in this case because it is a lot down below here I've got a couple of UCS fabric interconnects already configured just set up as a VP C up to each of the leaf pairs and I've got a ves already configured and deployed at the bottom here with a single VM which I'm going to be using for my testing so I've got the cpg at the bottom here which I've labeled global - EPG that's not yet configured that's something that we're going to do is part of this demo when we have it pushed from the multi-site controller and I will push that EPG to all on my sites the IP n again is a single device so going across it is the same device I'm using because this is in a lab but typically we'll see customers with an IP n device within each of their sites and they'll have some type of when connectivity or DCI between the various sites so in terms of our lab it is the same device but I do have separate connections on each side on the spine on the right side of course fabric 6 again simply X pine to Leafs single apec and on this side I have a bare metal host plugged in which I'm going to use as my other endpoint he's also going to be in that same target so at the end of the day here what we're hoping to accomplish is I'm going to do what's called a stretched EPG design and this is our most basic use case for multi site controller I'm going to be stretching the fabric the VRS the BD and the EPG across my sites and that's going to let these two guys these two endpoints communicate freely because they'll be in the same EPG there will be no contract required at this point it's a very simple setup in terms of the fabrics at each site now if we take a look at the if we take a look at my overall topology here one second we pull it up here is my the way I like to draw it out now in a very little bit of a higher level and this is going to show us how I've got my EPS addresses laid out so there are a lot of taps that are used for multi-site so I'll kind of walk through them here so we're going to need essentially for at least four tap addresses per site so on my left side here under fabric 3 I've got a first one is my multicast destination temp so if I need to send traffic to every other site that's a multicast address we're going to be using below that is my unicast address so if I have to have traffic sent to a single site that's where my destination tap would use the unicast address and below that we have a multi pod tap address and right now with a current implementation at 3.0 we're only going to be supporting a single pod we're not going to be supporting multi pod plus multi-site in this version we do have to provide this multi pod address but essentially once we get address our traffic directed to a site it will have to be further directed to a specific pod and that's where that EM pod destination tab will come into play but like I said for now because we're only supporting one pod we have to have an address within the same range and it will have to be routable over the IPIN as will all of our tepid ress is the last and the lowest kind of common denominator here is our control plane temp and this will be assigned for each spine in a site that's connected to the IP n so I only have a single spline plugged into my IP n therefore I only need a single control plane temp so I've mirrored my config on both sides I've just changed the middle lock text to signify the site so this is site three and this is going to be site six so captive very simple it really helps with troubleshooting if you're consistent with your your IP addressing system now in the middle here I'm going to use again it's a single device but I'm going to have a no SPF connection between each one it's going to be a point-to-point link so I've just got again the addresses for my point to point interfaces are just going to signify the sites that are connected to and it keeps everything nice and clear in my head now one very important note on the area I'm using area one as a regular area if you want to use area zero that's fine it'll be a regular area if you do want to use some of the other you know stub or not so study areas just double check any known caveat there were some issues in earlier code where using anything other than area zero as a stub or not so study area there were some issues with if you do say with a regular area no matter the area ID you're going to be fine 100% mine multi site controllers are deployed on a an external ESX box it really just has connectivity to the apex so on the APEC out-of-band that network kind of meshes into my multi site controllers reach ability between my multi site controller and my various apex in each site we're going to be configuring this as part of this demo will deploy the three VMs and we'll configure them as they were from scratch and I've got my my three node IDs I'm going to need here for my IP addresses from management that's it for the topology let's go ahead and get started okay the next step in our process is we're going to deploy the multi site controller nodes so I've gone ahead here and just used the ova image to deploy these three VMs now you can either use the ova image to deploy three separate instances or you can deploy one from the OVA and just clone it twice I've decided to use my name in convention like this the VM naming does not matter but the host name of the OS will matter so I've gone ahead and used these for mine I'm going to go ahead and power these guys up and we'll go ahead and get them configured so it just named them as you want node one two and three but I'm using MSC just to keep it very simple now you can move these three nodes across multiple hosts if you do want to have a little bit of redundancy obviously you know an H a or D or s cluster that's probably best practice you can certainly do so what we don't want is there is a stand-alone way you can deploy a multi-site but that would really just be used for a lab environment for any type of production setup we would highly recommend that you deploy these in a three node docker swarm cluster so I'll go ahead and open up my first console here to my node one and we'll go ahead and get him configured okay so it comes up so we're going to go ahead and login with our default credentials which is going to be root and Cisco I'll try it again and the first thing that's going to ask is to change the root password so we have to re-enter the existing password and you might get caught up on that but reenter Cisco again and now I will go ahead and assign my permanent root password okay it will have to be a complex password as well so now that I'm logged in here I'm going to go ahead and set up some the basic networking so there's a little utility you can run called MNT UI and this will allow you to configure the networking IP you can also do this through other utilities using if config or modifying the network scripts but the utility makes it you know kind of really simple to do so it's going to readjust this guy okay so we're going to edit a connection e zero and I'm going to go ahead and add I'm going to change the ipv4 to manual and then if I expand I can go ahead and enter the address here so I'm going to give them his address he'll be 204 and we put in the subnet mask like this I'm going to give them a gateway because ideally do you want them to be routable and that's all I really need if you want to add in DNS and a search domain you can do so but for my lab purposes here I'm not going to bother doing that a little bigger so we can see the whole console okay once you're all done you can go ahead down just toggle all the way down to okay and go ahead and back out and now we will have to deactivate and reactivate this connection so right now it's active I'm just going to toggle it now it's deactivated and reactivate it that essentially does and if config up/down just to make the new IP settings uh active now the other thing we have to do too is we have to set the node name now we start practicing the name here MSC we have it view a mandatory requirement that it has to be called specifically node 1 node 2 and node 3 so make sure your name needs very important if you don't will not start in your database will not work so a lot of the services are dependent on these names we're hoping in a future release that we do have some more leeway in terms of that naming convention but for right now they do have to be named node 1 node 2 node 3 okay all goods once I'm done I can go ahead and back out here and just as a test I'll probably just try and ping my gateway make sure I got connectivity looks good okay so I can go ahead and exit out of there I'll reconnect afterwards with SSH so this particular console is done so now I'll close him and I'll go ahead and move on to number 2 okay let's go ahead and log in again root and Cisco so these guys are not clustered in terms of you know their user accounts so you do have to set these up individually so I'm just going to use the same password assigned where you confirm it okay let's go ahead and run our utility we're going to edit e0 so I'm going to go a little faster here just because this is I kind of talk through them the first one it's really the same process so it gets a little tedious here and we'll give it a gateway okay and we'll back out and we'll be activate and reactivate and we'll set the hostname this guy will be node two okay and of course we'll just do a ping test let's make sure he can reach the other manager yes you can so let's good okay we're done here and one more route default Cisco Cisco change it to my permanent password let's run our utility and we'll do static 261 to the six so cat week zero six okay alright so there's our three guys just going to reset its connection and set its hostname this guy will be node three okay looks good alright sure we get connectivity x1 okay we'll go ahead and exit out there and I'm going to reconnect back now with SSH will be much quicker and easier because you're gonna have to do copy and pasting so from here on in we highly recommend you connect to them via SSH so we'll start logging through my new root password okay so there's node one I'll go ahead and get all three open here just to make it easy to move around there's two and one more okay so I've got my three here so I'm going to start off on the first guy where's where I'm going to run the initialization to configure docker so I'll go back over here to node 1 but you can really do this on any one of them we have to move to our folder where scripts are are contained they're not added to the path so we do have to migrate their builds and this will be our build number which we only have one build so that's my current build this might change if you're using a slightly newer build and prod H a which has the H a scripts that we're going to need so we'll go into here so in here I've got a few scripts that come bundle with MSE and the one I have to run initially is going to be this config one so I'm just going to run him and the C CFG and this is the knit one so we'll go ahead and run him so it's going to start creating the docker swarm so we're going to have three members one will be a leader the other two will be followers and we'll give us a second here so I'll just pause it while this is a doing a thing takes about two or three minutes okay so it's completed and what we expect to see is that the swarm was created the docker secrets are there here's kind of my handle my token I'm going to use on my there node so to join in a node that tells me here's the kind of the command I'm going to need on to join them so I'll go ahead and going to use that on my other nodes here so now I'm going to go over to my second node and I shall going to copy this token so if I grab this whole thing here and I'm going to have to specify the IP of the first node here so I'm just going to copy that one there I'm going to go over to my second node here and I'm going to navigate the same place so hopped Cisco MSC builds prod Ajay let me get the right build path now in here there's going to be a join script I'm going to run here so I'm going to run those join the script I can actually just paste it because it did give me the command from the first manager so if I just paste that didn't copy it let me try that again so I'm just going to go all the way take the token and everything we need copy that paste it in here and I have to give the address of my first manager so he was 204 ok so we go ahead and do that and it shouldn't take very long but he should go ahead and join the cluster okay doesn't take more than really like 30 seconds he for these now joined as a swarm manager so what I could do if I want I can go back to my first note here I'm just going to stretch it out for some of the outputs I can go docker node list LS and it shows me I've got two of my nodes here so I'm logged into the first node and the other guy is reachable so that's what we want to see my first guy is obviously my leader my second guy is active and reachable great so we'll go ahead and do our last one now I'm going to do the same thing on node 3 and we'll just copy the exact thing we used from here so same command and make sure we're on note three and give them a second to join so the only really problems you might have like I said if you don't name the nodes specifically node 1 node 2 node 3 you might see there's a problem or an error fault look on the manager when it's a you know you're a problem starting some of those services that we're about to deploy you'll you'll see some very obvious warnings okay so we've joined so even on this note here I can do the same thing okay so it shows me my my three guys are there they're all active and they are all reachable let's replay that to make it nice and neat okay so I'm logged into node three but these commands because it is a swarm we can run them on any of the any of the nodes okay so my node has been deployed my docker swarm is now ready for us to deploy the MSU stack now I can do the next part from any any node I want I'm just going to happen to use it do it from my node one here but I can deploy the MSU staff from anywhere so looking back into that directory here I've got this MSC deploy script and that's what we're going to deploy next here so I'm just going to go ahead and run this script so you want to give it some time here and make sure we very clearly can see everything coming up here so we're waiting for database services to be ready which is great and again keep an eye on these messages do you see something with not able to find a node or reach that could be a naming problem with one of the other nodes here okay so everything looks good so far to creating all the Kong the databases if your cluster and good ok so it looks like the cluster stack has deployed now the next thing we want to do is make sure that all the micro services are running so if I do a docker service list I'm going to see I've got all my various services listed here and how many replicas there are so some of these are going to be replicated across all three nodes and some are just can be local and you'll see they'll start starting up here so my MongoDB 3 here is he's ready to go MongoDB 1 is online I'm still waiting for the platform service and my UI so if I just kind of replay this command here they should start coming up here so I can see now my site service is replicated to two of the three sites and we'll just give this guy a few minutes could take about five minutes but I really like to stay here until all my services are replicated three ways now again if you see something like you know the DB one not starting for some reason that's typically resultant of the node name mismatch it can't detect it because it is hard-coded so get a couple more minutes here okay so everything started I'm just waiting on one more one more kong service to replicate so we'll give them a couple more seconds here okay let's check them again here so waiting for that last calling database to come up here so they can take up to about five minutes over here so we'll just be patient let's put the recording on pause until it comes up just in the respective time okay took about a three or four minutes here but I can see now that my Kong services started so at this point we're good to go we can go ahead and log into our UI so I'm going to go ahead and close these asset sessions I'm not going to need these anymore and we can go ahead and launch our UI now for the UI and just like the apec I can go to any one of them so we start off going HTTP and the IP or hostname of the respective node so I'm just going to go to my leader the first guy node node one and let them load up here so Firefox is caching my credentials but essentially the name and username this is kind of our front-end ng IX user ID so it's going to be admin and the default is going to be we1 c om e bang so it's going to ask me to retype it it's going to make me change my admin password so this is separate from our user our sorry root account so we1 c o m the bang and then now I can assign my new password and it tells me as I'm going through it if it meets my criteria for a strong password okay so passwords reset I'm logged into my GUI here for the MSC my controller status shows me 303 or online and available which is great now there's a bunch of other things we're going to have to do here but before we move on to configuring any sites or tenants or any of this I'm going to go ahead and shift gears and now do my IP n connectivity to make sure that's up and running here so we'll stop here and I'll continue on with the IP n connection okay so now that multi-site controller has been deployed and my status is showing that all three of the nodes are online and functional we're going to go ahead and configure the IP on itself so I will bring up my IP in here and this is just a nexus 90 232 device that I'm using here the only thing I really have configured in terms of kind of you know pre configuration is I've only set up the the policy map set up for jumbo frame so I've increased the system MTU to be 9216 nothing else is really done one of the nice things about multi-site is there's no dependency on the multicast within the IP n so I don't need to have pin biter support etc which is a requirement for multi pod so very simple requirements in terms of which devices you can use for your IP n now let's go back to Mike apology really quickly here so I'm going to have fabric 3 on the left here and fabric 6 on the right and both of these do connect the same physical device I'm just simulating my IP n through a single 92 32 device but typically see customers having either one or more IP and devices on either side or at each site with a win or DCI connection between them as long as I can route between the fabrics that's really what I'm looking for here now with multi-site we're going to need quite a few DTaP or destination tap addresses and these have to be routable through the IP n network so I laid mine out of how I'm going to assign them and this is part of what I'm going to do in the coming steps so I've got a multicast step I've got a unicast step a multi pod which is kind of like a dummy tap for our purposes but it still has to be assigned and then a control plane tap which is assigned to every spine and that's what we use to make our bgp relationships and tunnels between the various site spines so let's go ahead and get started here now I'm going to bring a mic apology for the site 3 so this is the connection right here I'm going to be configuring so I'm using port 1/9 and the interface address going to use is 3 3 3 2 and on the other side my spine connection I'll use 3 3 3 1/3 the only configuration I have to do is really the IP n-side manually the southern side or the side that goes to my spine and the fabric is going to be pushed from NSC so it's something that will configure later on as well so I'm going to set up both connections force 5 or 3 and for fabric 6 at least in terms of the IP and connection so I'm logged in here and the first thing we're going to do is I want to create a new VRS so mary BRS called multi-site just to keep my my multi-site routing this things a little bit separated next thing we'll do is configure the OSPF process so I'm going to go ahead and start that up go SPS and I'm going to call it my multi-site RM site I'm going to assign this to I'm going to send this to my VRS and lastly I'm going to go ahead and sign the read/write ages to be 3 3 3.2 which just happens to be the first interface I'm going to configure it but you're free to kind of use whatever Road ID suits your needs ok so that's done now I'll start configuring this first port here now one thing with multi pod multi-site is they have a dependency to use VLAN for so I will use a sub interface to do that and I'll set the encapsulation to be VLAN for that's hard-coded in the system that's nothing we can configure at this time so it's something we'll just have to you know just to consider as we're setting this up here so first thing I'll do is I'll go into intent in the config mode and we'll go into the first interface and the first thing I'm going to do on this interface is I'm going to increase the MTU let's try that again and I'll crank up the impudence what's going to match my OSPF configuration on the ACI side and the other thing I'll do is put it into the correct VRS and make sure we don't shut it now to create the sub interface we'll just go Eve 1/9 and I'll use sub interphase 4 and I'm going to use the encapsulation of VLAN 4 so I like to match up the VLAN and cancellation with the sub interface number and that helps me keep it nice and squared away same thing will set the MTU as well I do recommend adding a the description as well so you know exactly what your connections go to and if you ever unsure you can always use CDP or lldp to determine which interfaces you are connected to your notes here so we're going to add this guys well to the correct VRS and then now I can start configuring the IP addresses so make sure you configure your VRS first if you're going to do that and send to the sub interface because as soon as you start adding addresses and then you assign it to a VRS you're going to wipe out all the configuration as it's kind of details here so make sure you do the VRA first and then you do worry about the the addresses okay so let's start setting up the addresses so I'm going to go IP address 3 3 3 2 / 30 we're going to go ahead and set the OSPF Network type to being a point-to-point next I'll sign the OSPF in the area type so IP router OSPF process ID which is M site and I'm just going to put this in area zero and I'm going to keep it as a regular regular area type you could use not so stubby if you wish there are some additional considerations with that but from purposes of here we're going to keep it very simple and we're going to now shut it okay so that's my four going down to my first site so this guy here has been configured if I change my topology to look at six let's do this guy so he's using port 110 so I'll pretty much replicate most of the same configuration so I'll go into 1/10 I'll go ahead and set the vrf first so we'll go unto you 9000 quickly vrf member multi sites and make sure we get it and then we'll do the sub interface now and we'll do the same thing we essentially did to leave our end cap dot1q for MTU 1000 vrs member and site now we can start assigning IP so this guy is going to be 66.6 dot - slash 30 just confirming with my diagram we will do the iOS PF Network type and the OSPF Reder OSPF and site and we'll use area zero as well and of course do an OSHA okay so that's configured so my IP n is essentially done in terms of what I've got to do for it the rest I'll do on the fabric and on the multi site controller so next thing is we'll move over to the apex and do some work on there okay so what's that done we've got the MTN configuration complete the next part we're going to do is configure some of the a pic array or fabric specific configuration so the MSC will push some of those configurations so it'll push the logical node profile information for the OSPF but there is still some of the access policies that we do have to manually configure on each other respective fabrics so we're going to go ahead and jump into our first a pic I'll go ahead and get logged in here so the first thing I'm going to do is is set up those fabric policies so this is something again we have to set up just like we would if we're connected to bare metal host there's a domains a EPS policy groups and interface selectors that we would have to set up we have to do the same thing as well with the multi-site configuration so the first thing we're going to need is my VLAN pool so I'm going to be using VLAN 4 for the OSPF sub interface we have to allow that VLAN and I'll just call this VLAN pool M site very simple was a static allocation of just a single VLAN I'm going to grade it VLAN for so pretty straightforward ok next thing I'm going to do coming down I'm going to create the AEP and you can do this in whatever order you wish I'm just going to kind of go this way so I'll create the AEP next and I'll just call it very simply M site AEP and we will go next and I'm not going to send any interfaces next I will create my domain so I have to create an external routed domain will be a layer 3 domain and we'll just call this and sight l-3 the a epa previously created which is called m sight you select the VLAN pool and that's all we need for the domain config and that's something that we're going to have to select from the multi site controller when we get to that point so I've got the basic kind of global policies configured the pools and the VLANs the next thing I have to do is take care of the interfaces so I'm going to come into the interface policies and into policy groups and just like I would if I was using a bare metal host or something connect to a leaf I would lose a leaf policy this time I'm going to create a spine policy group because we are talking about a spine that policy interface and I'll call this guy here spine one polygroup link level policies terms of speed so I've got a policy that I've already created one with my ten giotto for my southern interfaces on the leaf and I've also got a policy called 40-gig Auto which is my IP n speed so the interface between my spine and my IP n is a 40 gig link so I'm just going to hard-code that I'm also going to turn on CDP just because I want to turn done and I will assign it to the a EP we previously created so I've got my policy group next thing I'll have to do is create a spine profile so if I expand profiles here and we will create a spine interface profile and we'll call this one spline one IP n and the interface for this particular spine and if I ever want to double check my work I always go back to my diagram and I'm going to be using port 532 this is a modular chassis so we got to know exactly which one we're going to use here so for the name of the selector I usually keep them pretty consistent with the the parent names and 5:32 will be the interface and of course my policy group for the spine has been assigned and we submit and last but not least we have to create a spine switch select policies so we've done this before many times we're doing leave selectors same thing applies with our spines so I'm going to create a new spine profile here and going to call it simply spine one and you can use the spine number you want or if you want use a node number you can it completely depends on whatever you like the really important part is the block this has to be the node number so my node for spun one is 201 and then we update that and it'll even give me the opportunity here to select the interface profiles for those spines so I'm going to go ahead and assign that spine IP n profile that we created over here just a minute ago okay so we're all done they're pretty much done that's all I'm going to have to do for this particular fabric now how to do the same thing on the other side as well so let's go ahead and do that so if I go over to fabric and my access policies and let's go in here so some of these I've already pre-configured just in respectful time but I'm just going to kind of showcase that I do have these already created to the exact same process of what I would have done on the other fabric so I've got the AEP there I've got my multi-site VLAN pool with just deal an idea for and I've got my domain using the same name these could be different domains it doesn't matter but I just kept mine consistent and then I've got my interface profiles also done so I'll just go show you the policy group there's the policy group and I have my spine profile called spine IP and one which will reference this policy group okay and then lastly is the switch policy where I do have that defined so there's my switch profile for the spine which is 201 I happened to call it 2-1 in this fabric but again the important part is that note ID under the block and that assigns the links it together okay so that's all good so the last thing I have to do now before I jump into the hardcore Multi multi site controller configuration is there's one last important policy we're going to have to consider and this should be done kind of at this point before we get into the multi site controller part so you're going to want to go to the intro tenant we expand the networking and protocol policies and we have to configure what's called our fabric external connection policy so if this doesn't exist you can usually create it so I'll blow this away here and recreate it for us just to show you the process so you just come down here go create intra site into your policy the only thing I really need to add here is the pod and our multi pod tap so the pod number for the first one is always going to be one we're going to keep that and the data plane tap we're going to use and I'm going to drive back over to my details and let me get that guy back up here it's rendered with me away from me is the first one I'm going to use is this Multi multi pod temp so it's kind of a dummy top because we really don't support more than one pod with multi-site at this point for the 3do release so but I do have to have an IP now you could copy you know some of your other tips but you know I like to keep it separate even though it doesn't matter at some point we will support multiple pods so I like to keep a separate IP for it so the one that's asking for me here this data playing tap is the multi pod tap IP so lv3 3.13 and we click okay and submit so that fabric connection policy now has multiplied tap this inter site data plane tap is essentially going to be our unicast step so this dot eleven IP will get pushed once we attach it as a site 10 multi-site controller so I'm just going to do the same thing for the other side I'm going to do this for site six going to my tenants into infra and then drill into networking and protocol policies and fabric external connection policy and we're going to create one and then keep one and this guy if I move my diagram over it's going to be essentially the same thing just going to be what six is so one seven two six six thirteen and we update and submit that okay so that's done now we can move on to the multi site controller and start adding in some of our sites now a quick note on the sites is be very careful when you're adding sites because there's something called the site ID that we have to assign that remains persistent for the life of the fabric so once that's been assigned you cannot change it now in my case here I did have my fabrics connected to this controller at one point and I've just unlinked them but the site ID I originally assigned still is applicable so it's going to read that in when I add my sites in your case if you're doing in Greenfield and you're adding sites for the first time it will prompt you to add the site ID or define it an Aspie unique per site so first thing I'll do here is I'll go to my sites and I'm going to add my first site so this guy I'm going to use this one is my first fabric called SJ fab 3 my controller my APEC URL and just going to put in the IP or a host name of your APEC and if you did have multiple controllers you can add you know number 2 number 3 4 & 5 if you have them in my case I'll they have a single controller on each fabric we're going to give it the credentials and make sure those are correct and that's pretty much it so even still I don't see any fields for the site ID and neither will you until you click the Save button now because again my sites have been previously added I'm not going to see that pop up what I will do is I'll overlay here a screenshot of what that warning message looks like for the site ID so I'm going to go ahead and click Save so it's gone ahead connected to that fabric and we've read in the overall fabric scores being 93 so we've got a good connection to it next thing I'll do is my next site so I'll do number two okay so this guy is going to be called SJ fad 6 I'll give it the year l6 880 here's your name password and again this guy was part of it so it's not going to prompt me for the site ID but it was previously set as 6 so my two site IDs respectfully where site ID was assigned was 3 to fab 3 & 6 for fabric 6 so that's been done now the next part is to configure the infrastructure and this is kind of a one-time as you're adding sites only this is really one time if you want to expand sites you might have to come back here so a lot of our control plane settings are going to remain default we're not going to mess around with the peering typewriting the initial timers leave those as is unless you have a good reason to change those there's also a no SPF policy which is by default so we set up you know things like the priority cost of the interfaces etc so again this is all default values I'm going to leave those but just make sure that you match up with your IP and configuration if you've changed knitting from default so the next thing I'll do is I'll come down under sites I'm going to click on my first site so this is the entire site showing me fabric 3 and right now you can see my bgp peering is off because I haven't really done that yet and this guy is not enabled for multi-site so in order to do so we've got to turn on enable for multi-site it's red in that site ID that is already with a sign and if you assigned anyone this would show up now I have to start assigning my multicast and my other destination tap addresses so let's go ahead and squeeze over here so looking at site 3 my multicast tap address I'm going to be using is going to be dot 10 so we'll go 1 2 3 3 . 10 my bgp AS is right in from the your system policy so that's already been read in you don't have to change that that's pulled in from the respective fabric the community is required so if you've got a community name you want to use or you can use some of the default ones I'm just going to use the unknown long as it follows the correct format it'll be fine the OSPF Airi ID I'm going to use will be 0 and I want it to be a regular type so you can also choose from not so stubby or a stub area depending on if you want certain you know prefixes linked into the IP n or not and lastly we need to set the routing domain so here's the name of the routing domain for fabric 3 that we previously created and I'm going to go ahead and select that now not going to click on submit next thing after I've got all these fields still doubt is I have to configure now the little bit of lower level configuration so now I need the unicast tap address so again this could copy with your pod IP but I'm just for my purposes I'm going to keep them separate so for my unicast IP I'm using dot 11 so this will be one seven two two three three eleven and then coming down I need to set my control playing taps here as well now this configuration I'm going to delete this and redo this because that was from my previous config so the first thing I would have to do is for each spine I need a CP tap or a control plane tap I only have a single spine connecting to my ipn so I only need a single CP tap address so this one's going to be dot 12 so I would come in here click on the spine that I'm going to start with which is spine 1 and the first thing I have to do is add the port now for my topology this will be port 532 the address and this is going to be your OSPF link so let me actually bring in my mic ethology here and this address here is going to be my other side of the OSPF so this will be 3 3 3 1 / 30 and get my fingers right here 130 the MTU I'm going to set as 9,000 because that's what the IP n has been configured for and I'm going to keep the default OSPF policy and save so now that I have configured that I've got the port about the MTU if you have multiple ports you can continue to add them I only have a single port but you can add as many ports as you have I'm going to turn on the bgp peering because I want this specific pine spine to peer with the other spines and the other fabric so I've got a VX land tunnel in between so now here's where I need a control plane tap address for each peered BGP member so my control plane tap for this guy for spline one is going to be dot 12 so once the 23.3 12 and I'm going to enable him as a spine reflector okay so that's pretty much it for that guy I'm going to go ahead and click on apply and we're giving a very glaring warning saying that I've got some overlap with some of my tips so we do do validation as well saying hey your multicast app and unicast pepper overlapping here which is not good so let's just double check so I've used dot ten for my multicast template to make sure that matches with what our IP n RS or what our details should be configured for and I'm going to take a look at my diagram and squeeze him over so dot n should be the multicast unicast B dot 11 so let's go ahead and make sure that's correct so there's my unicast tap should be dot 11 so just fat-fingered that so 10 so if I go back my site is dot 10 my unicast temp is 11 my control plane is 12 and then away configure it on the multi site on the back over here on number 3 was 13 so I've got 10 11 12 13 which has been assigned let's go ahead and apply that so it's going to validate it says yep everything's good now and if I go back over to that fabric now I can see that it's pushed this dot 11 address to the fabric so I know I've got good connection and we've assigned that unicast tap to the external connection policy so that guy is done let's go ahead and do fabric 6 it's the same thing here I'm going to enable him for a multi-site he's going to read in that his multicast up will be one seven two six six dot ten the a s and this particular fabric was using 600 no problem I'll use the same community for BGP and my area ID lu0 make sure I change this to a regular area and my routing domain is called M site on this fabric and that's good coming down we'll do the next one which is the data plane unicast up so 172 6.6 dot 11 and it could always make sure unique a step for this guy here is going to be indeed 11 and next thing I'll do pretty much all I need for that is that my control plane tap so I need to add the port first and for this guy we're going to be using port and I need my other topology which has my interface members on it it's going to be port 1 / 31 this will be 1 31 his IP address will be 66.6 dot 1/30 my MTU will be 9000 and I'm going to keep the default policy okay now we've got to turn on the BGP peering I only have a single interface so if you did have more again you could always add additional ones my control plane tap on this side for the spline will be 12 so one seven two six six dot twelve and rep reflector is turned on okay and we'll go ahead and apply that check configuration is successful so now that my config is good we can go back to one of our fabrics and check to see if everything's been configured correctly so what I might want to do is come over to my routed networks here I can see this inter site l3 out has been configured and if I expand it down here I can look at through the node profile there's four node 201 and all of this configuration has been pushed from the multi-site controller so it's created this node profile interface profile and on the USDF for that particular interface that I defined so here's all the information again anything if I were to change locally on the fabrics will remain in a fact but as soon as I go back to my mouth State Controller and reapply the config it's going to overwrite it so we we recommend that once a policy is owned by the multiset controller you let it remain so and make all your changes on the multi-site controller because there's a good chance soon as you reapply it or someone else realize it you could lose it so now at this point I've got the OSPF configuration good I'm going to go ahead and log back in to my IP n where I left off and let's go show IP ospf let's look at neighbors make sure they came up and look at the VAR FM site okay great so I've got a full neighborhood established between both of my routers so I've got the IP n port to the spine one fab three and my IP in connection to fabric six which are both up so my IP n is configured and as well as both in my fabrics so at this point my ITN configuration is done now from here what I would start doing is we'll start creating some of our schemas and our templates to push some of our policies okay so now that our IPM is all set up we can start deploying some of our MSC policies so we're going to take a look at the deployment we're going to be doing today so with MSC I'm going to be doing a very simple use case here which is stretching the tenant the VRS and the bridge domain and EPG across both sites this is kind of logically what it looks like it's going to happen so single vrf single BD single subnet and single EPG which will configure on MSE there will be some site-specific things we'll have to do for example if you remember my topology I have a VM on fabric 3 going through UCS and my other test host is going to be a nexus 3k interface on fabric 6 so we're going to have to define the endpoints differently than we will for some of these global policies we'll walk ourselves through that part so back I'm logged into MSC here we're going to go ahead and configure first of all our tenant so we'll log in here and we're going to add tenant and the first thing I'll do is give it a display name so I'll give it a name I'll just call it global TN I can give it a description if I want now the object name that I use here is what's going to be the object name and a CI now once this is set we can't change the object name however an MSC we can always rename this display name to a more friendly name if we need to so if you had a naming convention you want to keep consistent you can do so but you can always come in here and change it if you need to and I do want to have this the Senate associated to both of my sites so if I look at my spot book 3 here I can see that right now I do not have any tenant called you know global - Tien so that's going to be pushed from APEC or from the MSC to the APEC so I'm going to go ahead and associate with all sites in the default security domains and we're going to save that so we have to have the tenant configuration pushed first in order to push policies so now if I look I've got a tenant now called global pan which was pushed from the MSC and if I look inside here it's not gonna be really too much to it it's really just a shell of a tenant there hasn't been any bridge domains application profiles yet so it's going to be pretty bare for the most part so which just leaves who's expanded so the next part I'm going to do here is the important part of the meat and potatoes of setting up NSC policies and that's under schemas so I'm going to go in here and create my first schema so you can kind of think of your schema as almost being kind of application deployment if you will so my use case here which is going to be the layer two stretched EP g BD v RF that's going to be really kind of my schema so I'm going to give it a name up here and we're going to call it stretched EP g BD v RF and you can call us whatever you like but this is going to be kind of my use case I'm going to be doing here and under the templates is where I'm going to define the criteria that will get pushed to both sides so I'm going to use a single schema and a single template to accomplish what I want and that template because all the policies are essentially going to be the same I'm going to link that to the various sites so let's start off here I'll go template I'm going to rename this guy here and we're going to call this one we'll call it just very simple okay temp stretch policy and you can name this whatever you like now first thing I'm going to do here is I have to select a tenant so I'm going to go over here and I'm going to use the tenant I previously created so I could use the default common but I want to use a specific tenant so once I select the tenant it gives me all the same building blocks that you'd find on the apec so the first thing we're to do is create an application profile so I'm going to come over here and I'm going to give it a name here we're going to call this one app one very simple and I'm going to add my EPG here and if you remember from our topology here I'm going to call this my web etg and this can be stretched across so we're going to give it a name here for the EPG this is call it web EPG I'm not going to spend a gateway to the EPG level and the next thing I'll have to do is just find a bridge domain so I haven't configured a bridge domain so I can just start typing and it's at the bottom here I can actually create it on the fly so much like in APEC we can create policies I can do the same thing here so now I've tied the cpg to a bridge domain called BD one because I'm not going to use multiple ep geez I'm really stretching a single EPG from my use case and I'll have the end point on both sites communicate I'm not going to need contracts but I could configure contracts if you were doing into a EVG or inter EPG communication I haven't defined a V ref yet so we're going to go ahead and do so now it's going to call it VR f1 and the BD e and notice here I've got a red box around my bridge domain and that's because it's currently not associated to the VRS so chicken and the egg you can create policies just going to go back and link them up after you do so now the very important things I have to do for this bridge domain is I needed to be layer two stretched and I need to have inter-site broadcast unicast multicast traffic allow and that's going to allow that those policies to go right across from the EPG so if I fled the traffic from the site 3 it'll go across the site 6 and vice-versa now I also can define my subnets here so I'm going to do that as well and I'm going to give myself a subnet here one dot one dot one dot two five four and the slash twenty four and that happens to be the subnet that my endpoints are currently using and we'll go ahead and click Save okay I don't need any filters because I'm not using contracts so that's pretty much it I'm going to go ahead and save that the next thing I'm going to do is actually it's not been deployed yet so if I just have a quick peek back over here I can see nothing's actually been pushed yet I'm just saving my work as I'm going here next thing I'll have to do is attach this template to the site so which sites do I want to add this push to well I'm going to want to attach it to both site say three and site six now we're going to max out our sites per template a templates for site at five currently the first version but we will extend that to more so depending on how you want to lay it out usually one to three templates is pretty good to accomplish most scenarios and use cases so going ahead and save here so one shot once I attach it to the sites it kind of gives me some localized configuration under each of the sites so for example even though I stretched my EPG to both sites I'm going to have to do things like static top bindings and domain bindings to each respective site going to be done differently on site three I'm using a VM and on site six I'm using a bare metal physical domain which goes to my n3k interface so we have to do some things differently so the kind of universal or the global configuration happens under the template level then once I link to sites I can actually come in here I can do additional configuration so if I go and look at my fabrics here I've already got this VMware domain called fab 3 VDS configured and already pushed to the center with a host attached that's already been done here so I can see I've got my fab 3 VDS I've got no pork groups currently associated though so I'm gonna push the name from the apex downward there if I go over now back to my MSC I want to add that domain binding so if I come down to the web EPG once I select it below the sites I get kind of a site-specific configuration that I may want to add and the one thing I need to add here is the domain binding so if I go click on domain over here I can select the domain type which is going to be a VM M domain and it gives me all the other settings I would find at the APEC level the deployment immediacy etc so I'm going to select my domain profile sab 3 VDS I'll use on demand for the deployment immediacy and resolution I'll keep it on demand as well and everything else will pretty much stay the same so I'm going to go ahead and click on save now I haven't pushed a name on config yet the only other thing I'm going to have to do now is on the other side I'll need to do some configure as well so I'm going to go ahead and do that now now I will show you what I'm referring to I will bring over my topology so we just did this part here which was configure the VDS domain for my cyber three and then for fabric 6 my endpoint is actually hanging off of a nexus 3 case which i've just put it in there just to give it an IP just so i can simulate it being a host that's going off of leaf 2 on port 125 so we're going to have to do that configure here as well so if i bop down here and now to fab 6 go to the web BPG already highlighted I'm going to have to link the domain and then do the static path binding so let's start with the domain first so down under here under domains I'm going to add a domain take the type well it's going to be a physical domain because I'm really using my switches at end hoe so it'll be a physical domain and there's the domain of already created so that physical domain just like the VM domain on fabric 3 I pre created it I've done my access policies interface selectors that's all been taken care of already but it's allowing me to now use those domains as part of my MSC config same thing I'll go ahead and use my my - resolution and deployment immediacy go ahead and set those so I've linked that domain next thing I ought to do is set up the static path so I'm going to use a static port and the path type is just going to be a regular port I'm not using a VPC or a port channel lease number I'm going to use will be leaf number 2 and my path will be going to port number 125 so it's reading all these ports that are available from the a-tech into em SCS and allows me to select them now the VLAN I'm using in that VLAN pool that's associated with my n3k domain is just VLAN 50 so I'm going to go ahead and keep that encap and deployment immediacy we'll go ahead and keep this one immediate and the mode I want to use is untagged I'm not doing any tagging at the host level and I really want to just read in all the traffic that comes into the untagged with and go into my VLAN of 50 if you do want to do tagging on your host no problem you can either use trunk mode I'm just really using this as an host so I'm going to keep it as accessport and save okay so I've got my two bindings so now let's go ahead and click Save save our work and we can go ahead and deploy two sites so it's telling me that we have to deploy this to all these sites we're going to go ahead and click on deploy if there's any problems it will come back and tell me okay so I got checkmarks across the board that's great let's go have a look so over to my tenant on fabric 3 I should now see my application profile should exist great and my application EPG should exist there it is and below that I should have a domain link to the vmm domain and there it is great also I'll see my under networking I'll see the vrf and I should see my BD as well perfect let's just double check on fabric 6 make sure that looks good as well and down underneath same thing I'll expand the tenant just waiting for this little ketchup application profile up one application EPG there's the web EPG so I should see the domain list there so my physical domain should be listed here and the static port binding should be there as well so there's that pod 102 125 and that will be using VLAN 50 which is what I assigned here now just to show you the reason I'm using VLAN 50 is because that's a VLAN I'm using for my VLAN pool assigns the domain so we'll just kind of highlight that just so it all makes sense under my access policies I've got my physical domain here which was already created ln3 KF is and in here I'm pointing to the VLAN pool called m3k VLANs so if I want to blow that up I can have a quick look it's really just a single VLAN range so nothing much to it just because I only have the one house as my test so that's why I'm using dealin idea 50 on the static path binding have to exist inside of the domain that bounds that EPG otherwise we'll get false so you're a EP which is what binds your domain to the interfaces and that's where we've already got that kind of pre-configured I did that ahead of time so I wouldn't have to waste time with this trivial configuration all right so we've got that all done next thing we can do is let's see if our VM we can attach it to that the profile so I've gone back over here into V Center where I've published that EPG as a port group I can see here there's my tenant global pennant f1 web EPG and I currently have no interfaces so I'm going to go ahead and take my my web host here and I'm going to go ahead and attach it to the global tenant EBG now this particular host I'm going to open them up I got to make a quick change to his IP address he's currently configured for 2.2 address so I'm going to wet them over really quick so just bear with me while I love you that we'll do in a v6 no yes no okay so we're just gonna become 100 101 quite simple net match will stay the same and everyone needs to Gateway so we'll give them a gateway and his gateway really should just be one dot one dot one two five four okay all right so that's all done we'll go log back into the host now so why just looking at if config eat zero and I can see there's my address 1.1 to 1 so if I go and ping is gateway there we go now the reason that's working is because we already have the Gateway we've defined we pushed it from MSC so if I want to look at that gateway that was pushed if I go back over to that particular bridge domain so inside of here I'll have a subnet which I pushed as part of the config so I've got the bridge domain already in existence so once it comes down here I will have a bridge domain and my subnet has already been defined now if I take this off and I can very well do so if I happen to delete it I'll see that that will no longer work so now if I go back to my VM that's not going to work well what happens if I accidentally delete configuration well nice and simple I can reap uh from MSC so I can come back to here and go deploy two sites and that'll make sure that nothing got nothing got overridden go back here bingo there's my gateway which got pushed back and now I will be able to retain my gateway again great ok we know we got one side another question is can I ping to my other host ok okay so the interface that I'm using is just the wizz-eee 1/10 so I can see here I've got him assigned to the same subnet I've got him attached to a VR fm just called fab 6 on this particular host on this switch so what I'm going to try and do now is the same thing I'm going to try and ping from this 3k and sight 6 across over to my host which is in fab 3 under their sight so if I go ping one dot one dot one dot one VRS six okay so we are next pings go through as expected and the same thing here as well I should be able to ping back and we can hit them so it shows our policy work we didn't have to put any contracts now some verification you can do if I wanted to make sure you know if it wasn't working the first thing I'd probably do is go to the respective EPG s and make sure your end point being learned in the EPG so here's my test host here my test web DM I got the MAC address the IP address says I expect now because of the VM my learning source should be both vmm and learned learned being to see it on the wire I actually learned it from seeing traffic vmm tells me that I've learned about this VM end point from the center cells in the server and the interfaces it's currently using and then there's my dynamic deal an end cap that's used from the VM domain pool now if I look my other side if I go back to the tenant view I'll do the same thing I'll look on this side I'll take a look at the EPG drill down in here and all the way down to the web EPG and same thing I'll go ahead and look at the operational view and look at the client endpoints and here I can see there's my MAC address and learning from the nexus 3k there's the IP address and we are learning the traffic because we see it on the wire and it shows me that I'm learning this through this particular interface my end cap is obviously VLAN 50 which is the native VLAN for that that interface so we've demonstrated that we we can very simply configure almost all the policies rate from MSC here so very simple you know a single schema sets us up for you know a very simple deployment if I want to have a look I can always look at the health from our dashboard so from our dashboard view I can see if there's any problems or you know perhaps you know a policy failed or you know there's an issue from this view rate in here I can see that the schema shows me that that particular template is good to go on both of my fabrics so it gives you a good nice snapshot of the current health that's of that as a quick recap we did a whole bunch of things here today so we went we did a deployed a stretched fabric across our sites which worked quite well we configured the IP n so we brought up our our fabrics here we attached both of our sites to the MSC after we did the installation very simple configured our IP n a very simple process but if you haven't done it hopefully seeing what I did there does make it a little bit easier when you attempt to do it to find all the tap address ISM again I highly recommend we do so these tips are very important just to kind of cover what they do so my multicast tap would be used if I have to communicate with multi sites so almost think of it look at any caste for all your sites the unicast is if I have to hit endpoints within a specific site or all endpoints within a site my M pod is going to be specific to a pod which right now is kind of irrelevant since we only support the one pod that's the only one we're ever going to have but as we expand support to include multiple pods you'll have more of these M pod D tabs assigned and then the lowest we've got our control plane tab for each of our spines and because I only have a single spine in Mike apology we only need that single CP tap per site so ladies all out keep a good network diagram that way if there are any problems we can very easily refer back to this and say okay yes we're learning our tunnels on the correct interfaces everything's looking good gives us a really a good road map to look at your environment so hopefully enjoy the video I thank you very much for watching

Channel: Robert Burns

Views: 25,955

Rating: 4.9459457 out of 5

Keywords: ACI, Multisite, MSC, APIC

Id: HJJ8lznodN0

Length: 74min 56sec (4496 seconds)

Published: Thu Aug 17 2017