Cisco ACI Fabric Forwarding

the idea of a pure AC i network now an AC i network can integrate with anyone else's network that already exists you can have servers sit anywhere you want you could have one of my competitors which is in place we can still do policy automation we can still do the grouping we can still do what I showed up here all we really need is an 802 dot1q trunk or we can pair up BGP lsbf and the rest but the beauty of the hardware of a CI if you were looking at a full a CI deployment is if we did it right the hardware networking switching itself disappears we have what we have what we call a zero touch bring up you plug a controller into the first leaf switch it discovers that leaf switch then discovers the rest of the network it discovers all of the topology or you can give it a cabling diagram to say this is how I want the topology to look verify that it's connected this way it will then actually IP address all of the all of the network devices on a V tap virtual tunnel endpoint we use VX LAN solely internally within the system so let me draw up how this looks I'm going to draw a let's say for spine leaf design and I know I'm like an artist on the whiteboard here this is just pure beauty so spines Leafs spine leaf design right when we look at the way in which we do this is everyone familiar with the advantage of the spine leaf design or why we use this finally design does anyone want to take two minutes on that good no so your network engineers soon all right so we have our spine leaf design we look at this as two spaces we call this the infrastructure space or infra and we call this the user space user not necessarily meaning your users users meaning anything connected to the ACI fabric so all of this is an ACI fabric internally each one of these is a virtual tunnel endpoint and in fact in many cases they have more than one virtual tunnel endpoint on each device depending on what it is we're doing but we use VX LAN to route in between everything what this allows us to do is give you 16 million segments instead of four thousand VLANs so we can do very large multi-tenancy we can do large-scale amounts of VR F's for overlapping subnet what comes in from the user space is irrelevant to us internally so VX LAN here this can be VX land sorry I swear I used to be able to what right whiteboard in V GRE dot1q or untagged right so what comes in is irrelevant to me I'm going to strip the header if there's a header on it I'm going to put it in a VLAN header move it across increment the packet counter one time as if it hit one routed hop and drop it off wherever you want it when I drop it off I'll drop it off however you want it to drop off so again we want to make the network transparent call it network normalization if I have a hyper-v hosts connected here and it's using NV GRE and a VM VMware host here using VX land and I have a bare-metal server here using untagged frames or dot1q I can move the packet five about five microseconds to any port to any port and I can rewrite from MV GRE to VX LAN to an untagged or dot1q frame as it moves forward because you told me this group connects to this group you don't have to configure all of that and it's transparent to the end device out here the end device out here does not know that we wrote in an RV excellent and moved it out if you were running for instance there's a very big virtualization company with a competing network virtualization product you may have heard of them if you were running that on top of my network I can provide advanced network visibility into what that's doing and I can take their VX land I'll translate that frame I'll use a different vena d'Or virtual network ID internally and then when I send it back out I'll put their VX lamb back on it and hand it to the device internally we leave it all normal and we don't do double n cap because double n cap waste bandwidth so the VLAN example is compelling because a lot of customers are already using that it's nice to be able to just say this this VLAN is it represents this network that I already have set up just put it in and put in an endpoint group and you're done the the stuff I think above that is is where the complexity comes in because some customers have already gone to the X land in a multicast fashion just because they've they've needed to maybe they're using 1,000 V and it's a simple way to do that right I think the one thing that I have yet to figure out is how the fabric and handles that state being transferred from endpoint to endpoint outside the fabric so multicast be excellent has to figure out what max go to what V temps right basically here's the veena that I need to use normally those those multicast frames go through a fabric and they find each other right well a CI strips all that off so what are you doing to communicate down to those endpoints the state that they're looking for is that something that you do or is that treat it basically as just a bum frame no what we'll do is pair with that multicast as it comes in so we build multicast trees within the fabric we could actually do multipath multicast and fast reroute for multicast failures but we'll take that multicast stream and deliver it to the end devices it transparently the same way that any other network would use them we don't make any change to how that actually works so even though you're still stripping off to the ex line because you have your own internal way of doing that but when it gets delivered the same thing gets appended and that discovery process can still take place exactly I'll know which Leafs are participating in that multicast tree and I'll drop it to that leaf that leaf will then know which ports it needs to drop it to so to build a multicast apology to know that and be able to deliver it back out as if it in the same way multi-class works today or your external multicast the other thing we do is our controller itself is the exact opposite of any other Sdn controller out there any other Sdn controller out there focus in life primarily is forwarding telling packets where to go open flows the best example this it's using a five couple or twelve double match to tell packets where to go program a route table not a fair statement we don't look at packet forwarding this topology comes up it uses is is to build a router topology from any v10 TV tab basically any leaf to any leaf our controller itself never sees a packet never programs around it is never involved in the packet process our spines do some magic our spines can handle in a hardware database up to a million ipv4 or ipv6 endpoints the addressing for them this allows us to eliminate broadcast unknown unicast and multicast on the fabric at line rate in Hardware without ever hitting a software controller to do so so if I have a broadcast say unknown unicast hit this leaf if this leaf does not know the destination it sends a unicast message to a proxy on any one of these spines that spine at line rate hits that hardware database which can be up to a million entries and in fords it unicast to the destination drops it off let's say it's an ARP drops it off when it gets dropped out to user space nothing's changed from that art that art reply isn't unicast ARP reply just like it ever would be when it comes back this leaf relearn's we call it a zero zero penalty miss if my leaf does not know where it goes at line rate unicast it learns it sending it through the spine proxy there's a lot of things that we do in the hardware that enable what we're able to do on on how a CI works and becomes transparent underneath you blew through that fast um and missed up because I didn't want you to ask any questions no about I was just trying to take a call that in in one shot so the spying only know about V taps endpoints two endpoints right yes okay and so their routing table all stored in hardware is that where the merchant plus comes in yes the merchant plus piece comes in there so we have basically fabric cards on the back of the spine that are all based on a Broadcom chipset so the fabric cards are based on that on our spine line cards that is the merchant plus that is the Asics developed by the NCMA team now I NS bu so those are the ones that are able to allow that million address database to be built and allow some of the other things that we do within the fabric so basically the idea is here we have a proxy route database that knows how to get any ipv4 or ipv6 endpoint to any leaf destination to the V tap it needs to go to which would be the leave and yeah and clearly the spines are ignorant of what's outside of the V taps or what's been encapsulated it doesn't need to know that's correct yeah that's right okay you would mentioned the a pic does not program routes he doesn't do any forwarding doesn't know the fabric figures that out for itself how does it figure that out again you heard is is and they're real quick and so is is is the routing protocol between virtual virtual tunnel endpoints within the fabric and if it does the automatic learning via basically lldp so automatically learns then it uses a DHCP pool which you can manually configure allow it to auto configure to assign out addresses on each of these links between V taps and then that route apology comes up based on that so it's it's own little distributed control plane that's self-sufficient and discovers new additions to the fabric knows that a new leaf switch got added our new spine switch got added lldp discovery he knows where he's at in the topology is is is used to learn all be all the beat apps all those IPs for the beat apps and that will scale up to a million entries in that the merchants the plus silicone depending on which spine you're using yes okay and so that's so it is self sufficient to get traffic beat up to V tab outside of the controller the controller is just policy that's right so if you think of an open flow controller an open flow controller tells packets where to go yeah a pic controller tells packets Windigo so you do apply an application network profile I already know how to get that packet where it needs to go you're turning on the connection that makes sense yeah now the other thing the spine does is obviously we don't work in isolation so RIS is royalty topology and VX LAN here is complete completely transparent to the outside but somewhere here you're going to have a router right so that router can pair up to us using bgp OSPF or static routes if you want to have a lot of fun what happens is that once that's pairing once that's paired the leaf sends the information to the spine the spine becomes a route reflector so it puts those routes external routes into its hardware database as the proxy and then put it and then they get learned by the Leafs as they're used that would let me a device here know that he needs to send it to this or she depending on the sex of your packet to this leaf I get it out of the fabric so you said something important there the in a certain sense the way in a CI fabric forwards traffic from leaf to spine isn't actually important it's interesting but in a certain sense we don't care it's a cloud it we know we're going to the the traffic is going to be delivered we don't have to configure that it's not it just happens as a part of those switches being an AC i mode absolutely so I always describe it I'm probably gonna get killed for saying it on video but I always describe it that a so Tom Edsel was our CTO and co-founder Tom Edsel is one of the best Network ASIC designers in the world he's been doing it for Cisco for years any product you use from Cisco he probably touched the ASIC if Tom did his job right and if our hardware team did their job right this all disappears and you focus on deploying applications that is exactly what we're trying to do we want to make this disappear give you all the visibility into whatever happens when something isn't working but we don't want to make you think about okay this is VX land how do i bridge it and Gateway it over to MV GRE and then get it back out to this each one of our ports is a line rate stateless firewall and gateway device for MV GRE VX land VLAN untagged ok so now this puts some context around something you said very early when you started this section which is APEC is still useful in a non Cisco hardware you can still do things with competitor switches can you just say that again and what you can do and can't do yep we can work with competitor switches so let's say let's say you built this design but you had some existing and I'm not going to name any competitors but because their competitors I'm gonna draw them in red because no one wants them you can hair up to that switch you can hit you can have servers hanging down here and servers are always in 3d you can have servers hanging down here what we care about is that 802 dot1q packet header so if you have two VLANs coming in here VLAN 10 and 20 D LAN 10 and 20 can be okay so we're coming into an AC I fabric now yeah you're saying okay you're gonna pick to the actual container what if that's not your fabric what can I pick do if anything today today nothing today you do need a small subset but I can actually draw that integration of so this is one of them if you have you wanna actually push policy or something right if you can do in OVS in the future what would stop you from running hypothetically there's a vendor out there today that runs OBS and hardware to using op flex down to that top rack switch I'm sorry a pic down to that top rack switch so this isn't this isn't out yet it's still being worked on but the whole group based policy stuff in open daylight it's very compelling and it's basically aimed at that kind of thing because it's the same policy model right not much different but it is aimed at working with other open platform let's say you have an existing three-tier network and this is from vendor exit is not cisco equipment or it is cisco equipment it could be any of the nexus platform to be catalyst by former could be from any vendor this is your three-tier network today correct which layer where do I enforce policy on this network today today's policy is complicated where would I typically enforce policy I'm giving you a hint egg tag right egg is typically my layer three boundary right and a lot of times we have a maybe you know in the Cisco world this would maybe be a catalyst 6500 with services modules hanging here at the act adieu firewall load balancing that kind of thing maybe service appliances hanging off here at the AG to do that so packets go here for policy enforcement they go back to wherever they're going or they go out is that fair so the traffic pattern you have today is traffic comes in and I'll sorry let me comes in out down for policy or it goes up out down for policy that's fair okay so aggregation is your policy enforcement boundary if you want to go to a CI on that network and you want application network profiles and you want service automation across layer fourth or seventh service devices virtual or physical that kind of stuff what we basically build and I draw it this way this is not how you'd physically do it by draw this way because it makes more sense in our minds if you would build a small spine leaf architecture ACI spine ACI leaf ACI spine ACI leaf 40-gig links between knees and then 10-gig links connected to your AG you make me your gateway and now guess what packets do so this is a dot1q trunk to your existing AG you make me your gateway now I see all of your packets that require enforcement and now I can automate the service appliances you can hang them off here you can leave them hanging off here if they can be hanged at the edge they can be virtual sitting in the hypervisor so basically you take one hop to me I use a dot1q tag to put you in a group and I integrate with this existing infrastructure does this make sense you just need to make sure you get through your spine to enforce the policy you make sure the packet gets to me with whatever that device configuration tool is on that equipment as long as the packet gets to me I can identify it in a group put in an application network profile enforce policy the only production change you have to make is dot1q trunk and then move your S VIII or your default gateway over to me okay now the next coolest piece is coming next year we'll also be able to do what we remote leaf so you could have a nexus 9000 based leaf hang off this existing access layer or off this existing aggregation layer and still be a part of that fabric and that gives me full policy enforcement for all the hardware acceleration stuff I do here but all I need is layer two early as we're not working between so this is for integration edit not really integration and maybe overtime migration depending on whether or not you're doing that so we're definitely not designed to be a full like rip and replace your network because there's like no percent greenfield environments in the world right sorry Joe with regards to the leaf can you I don't if you can tell us anything yet when that's released how the leaf connect back into the fabric if there's something else in between light you know some 65 hundreds or whatever so yeah you're things like lldp not gonna work I mean is it going to be some kind of like a TCP session between that leaf and the spines or all the Leafs further up basically it will be communicating directly with the spine via V excellent to whatever the connected switches okay and that's how we'll know what's there any other questions on this it just asked a question similar to yesterday to if you have this you know for rack scenario and you know a common VLAN across multiple racks or common else.you domain you know where is that default gateway exists for a single segment we're back to that I actually think it's I think this will be an easier answer here yeah I would expect so let me through that again I'm saying so if you have multiple segments right across multiple racks and you want to do enter segment routing it's reveal and routing you know you know where where do you perform that that that routing functionality which would fall 22 segment's where's the default gateway exist oh what's the word distributor default gateway whichever leaf you're attached to is your default gateway okay it is you know yeah conversation over yes yesterday was like I did you danced around for nothing right now yeah it was his 10 I think was a the optimum number of runs yeah so whatever leaf you're connected to we do it should be a default gateway that will make your routing decision the fabric itself looks like a single routed hop that if from a packet perspective it looks like a single rabbit hop well when you say that to troubleshooting becomes a little more complicated in that it's how what do you guys do to get visibility into what that looks like now because now it's a black box right what are we doing that we can see and into that and where the packets are going in plate troubleshooting we don't need to do troubleshooting you kid that's full gin or whiskey whatever was um so AC I never drops packets ever it's perfect so you don't have to troubleshoot no we do a lot of advanced telemetry within the fabric so this is gets into a bit longer discussion but we have about five minutes we'll go into it within the ACI fabric we have what we call well we have several things but we call it a telemetry basically its network visibility stuff so if this server is an FTP server and it's dropping packets that's pretty easy to figure out on today's network right if it's dropping packets how easy is it to figure out where those packets are dropping on today's network that an easy task complex tasks and impossible tasks to do a tedious task yeah yes tasks okay so tedious tasks same thing same same idea so that's one of the things that we've done here we can turn on what we call a Tomic counters for either counting packets or bytes if you tell me count ftp packets I can see that a thousand packets hit this switch that they were multiple that they were multi passed across all of my spines they took four paths I could see that 250 hit each spine just for example and I can see that this leaf only received X number so I could instantly in real-time freeze-frame snapshot and say this is where every packet hit this is every link they took this is where you're dropping your packet when you start to look at an application we can give you we do health scores so when you if you took Microsoft Exchange and you put it into an application network profile I'll give you a health score Microsoft Exchange Microsoft Exchange is a hundred percent healthy on your network you can give that to your exchange admin visibility into that havoc or that shows a hundred percent healthy I call it the knotted button for network engineers because we're always at fault right that health score is comprised of latency packet drop throughput in real time our packets are actually tracking applications not exchanged isn't working they're still going to blame me you know you can have a health score low say well your health scores wrong right exactly exactly because there really is that you know you have to you have to dig deep to prove it's not me and if you know I mean I think even from a network perspective we kind of see this is a black box you and if the application owner every understands that it is a black box that's that's where they point every time right so how do you validate ice ices organ properly exactly do you expect engineers to learn is is probably not but if they did know it would it help them so do have access to those commands in the box if not like other tools to just you know validate ice ice or validate l2 l3 things like that yes you can move through the GUI the CLI or the API you can validate any any component anything the system is doing absolutely everything's exposed from our API on top of our API we built a CLI in a GUI not the other way around so you could write your own GUI to do inspections you could write your own command line to do inspections we actually have some stuff like that going on actually Paul's doing a lot of work for some different things in the way in which we're looking at translating commands from other languages but you do have visibility all the way down into how the topology is built you have the ability to dictate what IP addressing is used how that topology looks and what the cabling topology is and you're able to troubleshoot that we put in so again some other advanced tools that let you pull that information out I don't like the caller black box cuz again it's switching and everything standards-based but yes we have put into that in that scenario you would never configure let's say 9k directly totally get it but in terms when you go on to that switch could you do like a show is is database or whatever the command is to pull out data yes yes yeah and we can send out SNMP and syslog messages and all the things you would expect for normal network management tools you are able to get down and see at a switch level what's coming up so you can you can go down to that switch level look at what that box is doing it's just exposed as an object showing you the attributes it says sort of lacy there's no sort of mentions like how that data is actually exposed you know there's really counters stats whatever it's all there so in how do I get out there so it think we Paul would probably walk through some of that into sessions he's doing is he's good he's gonna actually show a little bit of this as it goes through that's how that works make sense any other questions on this I thought there was bgp and within you know the back end or the black box a few of aha but there was mentioned a BGP in there today was there um so we were looking at different things for how we how we do our control plane protocol internally and that would be one of the things that would be capable of doing that so there's no today let's say shipping is no BGP it's three eye size we can put my pair BGP outside now I mean with it I mean within the fabric though within the fabric is is is probably the better it is overall the better protocol because the scale and the fact that we're only hitting that many V types the speed which is is works and some of the other things but know that we don't use BGP to route internally okay is there anything I'm missing on that pole of BGP use cool have a Twitter question and how do you intend to get trust SEC and a CI to work together there are there's a project that has been submitted to the IETF by us and a few other vendors called network services header nsh you can find the draft basically it's a context aware packet format that allows true service chaining and then the ability to work with other devices that are currently doing security and the rest so over time you will see a convergence of these products and how they work but that's all future roadmap and I call it I'll call it vision by vision I mean I make no legal claims to the fact that this will ever exist another Twitter question where the Cisco live where the ladyship and opendaylight lisa Cawood so a cisco is very aligned with open daylight we are working very closely with them within ACI we are also working very closely with them on the group policy project so we are still very much a proponent and push of open daylight it is another way to do this the Nexus 9000 in nx-os mode would be able to run with open the opendaylight controller and use open flow as a protocol

Channel: Tech Field Day

Views: 17,881

Rating: 4.8857141 out of 5

Keywords: Tech Field Day Networking Field Day Networking Field Day 8 NFD8 Cisco Systems, Inc. (Business Operation) Joe Onisick Application Centric Infrastructure ACI

Id: bJ7UuZ2-64A

Length: 25min 27sec (1527 seconds)

Published: Thu Sep 11 2014