Cumulus Networks White Boarding Overview with Pete Lumbis

and for those of you don't know me I'm cute lumbus you can virtually harass me and I won't see it at attack sorry not attack anyone heat CCD on Twitter I work at cumulus networks I'm a technical marketing engineer these days I like to say I do everything but write code for the most part and what I want to talk about today you know here with Mellanox here with Ixia is is where cumulus fits into this model and kind of how we see what these two organizations are doing how they mesh with what we do and and who we are and what we do and then show you some of this stuff and a little demo so for those of you that don't know what we are is where the Opera where the network operating system sitting on top of the switch so I take a Mellanox switch and on top of that switch right this gives me my my ASIC in here from Mellanox that's the spectrum a sixer and we are the software thank you running directly on that bare metal switch so we're not a VM we're not an application on the side we are an operating system running directly on that switch writing device drivers spinning the fans flashing the lights programming music so although we are a software company we have super deep knowledge of both the hardware and the platform the optics and the ASIC itself and so I think it's one of the most important things is is we are a software company that really really understands the hardware very well one of the things that's super cool about cumulus Linux is we are just Linux we are just a Debian Linux distribution so we've we've added some patches we've made some modifications everything that we've done that we are allowed to open-source and send upstream we send upstream so one of the perfect examples of this is VR s so Vieira's didn't exist in Linux a couple years back and we looked at some of the options that existed in Linux to do this some people are familiar with network namespaces and we looked at Network namespaces and we're like this isn't the way that you should do routing table isolation you know it doesn't work so what we did is instead of just building a vrf and doing it ourselves and forever and ever it will be the cumulus PRF solution before we wrote a line of code in cumulus Linux we went to the Linux networking community and we said hey we think that we want to do something that's not Network namespaces it's called the PRF pretty common in the network world and it's totally foreign to the server world and this is how we think you should do it and here's the code that enables here us on Linux and through that open source process we got our standard or our proposal accepted they took our code and our vrf patches are now part of the Linux kernel at that point we went implemented on our switch so the way that you configure a vrf on cumulus linux is exactly the way that you configure a vrf on a Linux device the way that you configure interfaces on Camillus Linux is exactly the same as you configure it on a Linux device well what's really cool about what we're doing is we look at how do I build systems data centers architectures networks using simple building blocks picking those LEGO pieces and turning it into a Saturn 5 and so I can take Linux as a building block and I can take an application on top like free range routing the open source routing suite that we use which was a fork of quagga this is where the e VPN functionality lives is within free range routing it's the part doing BGP speaking evpn to everybody else doing the interoperability and I can take something that career and routing number one I can open source it send it upstream I can run it on servers now I could have my servers doing BGP to my top of racks which we support something called BGP unnumbered so I don't even have to put IDs on the links so we're gonna have my servers doing BGP peering to two or three top of rack switches I think about like an nvme scenario we're talking about earlier I might want more than two top of rack switches to really maximize bandwidth and scale out well I could connect to four top of rack switches without using an M lag or a lag in that scenario but we can take something like prearranged routing or whatever the again no simple building blocks are and we can put more building blocks on top of it so we've created a CLI called Network command-line utility and June showed some of that off so for folks coming from the Linux world they know IP link IP show they're familiar with those Linux commands but for network engineers like myself you're coming you're not coming from that world those things are hard to understand foreign some of the documentation isn't exactly great and then you go looking for it and you're like hey how do i do something i can't find information and the answer is RTFM and you're like but the the MIT and the M is terrible can't read it if it's bad so we created something called Network command-line utility and what the network command-line utility does is it gives me a CLI another simple building block on top and if you look at some of the other vendors out there they're doing Linux things and their CL eyes once you enter the CLI you've gone through the looking-glass and you never come back right I can never real Enix eyes my box once I've gone into that or I start and CLI land and as soon as I go into Linux land the door closes behind me we don't want to do that right we want again when you use this as a simple building block so when I come into NC Lu and I type net add BGP autonomous system number what that actually does is write into the configuration for free range routing and when I do net add interface port 110 one one one that writes into the Linux Etsy network interfaces file yeah I really appreciate that yeah that at least somebody is still thinking at the non-developer so thank you and so what's important about that is that I as a network engineer can come through use the CLI get tab completion get in device help with net help and see documentation of how to configure something including like M lag where it's multiple boxes I can configure it I can walk away my sis had men who's on call that night and then look at the Etsy network interface this file I know exactly what I did and this also is really important because I think every single vendor you've talked to this week has been hey check out our box and check out this application you can run on it well the problem that you always end up seeing with these I mean the the the oldest story there is just automation is that hey I can run puppet you can't use anything puppets ever done before or ansible has ever done before but we built like three things and you can use those three things with that application well because we're not changing the model I can come in with something like ansible or I can use sensu for monitoring or I can use Telegraph on box for streaming telemetry and then I can come in as an operator when things don't work and I can use my CLI and have that shared language which is still Linux how I interface with Linux is up to me and again it's about those simple building blocks being able to be stacked on top of each other to make a more complex system that does what you need it to do so any questions about that so overall you know what were what we look at is is Linux is kind of the model that everybody's rallying around it's the way forward and everything that we do that deviates from Linux is a problem and so what that means is that before we can get down to the switch ASIC we have to make sure it works in the software and we look at the software as the model for how we do everything so a lot of platforms other vendors other other products when you do a configuration some of that stuff starts in software and gets pushed to the asic some of that ends up side loading and coming straight to the ASIC and totally skipping over that software component so because we have a simple building block of Linux I can take this whole component and I can remove just my software piece and run it and what's called cumulus BX virtual experience so it's not a view router it's not a V switch it's just a VM of our software model and it looks exactly like this which it runs on top of and again because we're putting everything here first everything that works on this Mellanox switch works on my software so I can configure VLANs VX lands and ACLs and do e VPN you VPN bridging EVP and routing symmetric mode asymmetric mode I can do it all in a free VM and that free VM again instead of trying to do something different or trying to say like well you have to run OpenStack to stand up two nodes we once again wanted to take simple building blocks though cumulus VX Enron and VMware virtual box or ABM so again we're taking simple well understood constructs of these VMs and building more complex things and now since I have a licks device a Linux operating system running on a virtual machine that has lots of Linux operating systems on it I can now add in a tool like vagrant which allows me to describe the connectivity of devices and stand these things up and so what I can do here is I can take tens hundreds of nodes I was working on a lab just last week with I think a hundred and eighty seven devices in that virtual topology what I'm gonna show you today is a little bit smaller than that but still sizable and what that means is I can take the exact one-to-one replica of my data center with the exact one-to-one features in my data center with the exact 101 configurations in my data center and I can start modeling configuring predicting testing validating training learning with them and I can use this tool vagrant and I'm not here to teach a vagrant today but just to let you know and do something like vagrant up and launch those 200 devices come back a few minutes later and it works and we don't have to build anything for big rent we don't really have to build anything on top of this one of the other things that's also amazing is KVM added space as a hypervisor allows you to do tunnels between bare-metal servers running VMs so I can have a VM on server one at a VM on server two and using a UDP tunnel they look like they're attached together with an IEEE provider so it's running 200 devices sounds great if I've got the hardware for it so again I want to take this simple building block and scale it out and build a more complex system so I could take two or three or four servers and spread those 100 200 300 switches across them and virtualized whatever it is that I want to build and again it's about those simple building blocks that continue to whether it's a spinal Farkle texture whether it's a Linux model whether it's a VM or whether it's virtualization and that's really kind of how we see the world and then once we have that viewpoint we can do anything from there I'm sorry if you've already much sense but the cumulus px is just testing just for isn't it it's not gonna replace like a V router not a V router so we've done approximately zero work for packet acceleration so you can ping you can trace route you can send data you can run iperf on it if you want but we haven't done like a DB DK layer we're not doing like an FTO layer it might work but we don't care about this it's like a validation platform it's a validation platform okay and I and I will tell you that even you know customers that work with our consulting services what we actually do with every single one of those customers as we build their entire data center in cumulus VX we build all the automation and key mosaics while we're waiting for that gear to show up and that whatever that lead time is and then once the gear is there we plug it all in and just drop the exact same configuration everything on there it's ready to go all your codes ready to go a powerful statement so cumulus plus VX is just the VM so there's no hardware component just not just VX but like oh can I use the mill adducts which gonna be some other white box so we run on roughly 70 different switches today from I think I'm gonna get this number wrong I want to say eight different manufacturers because we are the bare metal where are we the operating system for that bare metal we have to know how to do things like what is the the order of shaking the +5 volt and minus 5 volt lines on the CPU to indicate that the switch is being turned on and so we have to write those device drivers and understand that model so we can't run on just everything you know you can't go by like a neck here figure out how to how to like crack it and install cumulus on it we do have to be able to write to that platform to be able to understand it this like any other operating Thanks do any other questions alright so now I'm going to show you some demos stuff and if we're real lucky it'll even work invoke the demo gods Yeah right so what I'm gonna show you is built entirely and cumulus VX it's all in github github.com slash Columbus slash and ft-17 it's the pointing at it that makes it work so I have all idea looks kind of like this so again I'm mixing but I'm using those same those simple building blocks they have 32 Leafs I have four spines I'm running ebgp in this fabric and I've attached six of them to house in my scenario two of these hosts are using vehicle and between them and the other four are part of a docker swarm cluster those they said we believe in these simple building blocks we believe in building blocks based on Linux so what we've done is we've created an application called NICU NICU is an agent that lives directly on my switch and it listens the Linux net link messages as well as a few other things and so there's a lot of vendors that talk about pub/sub models and things like that Linux was the original pub/sub model so I can write an application that listens to messages within that kernel and then just pick them up as they happen in real time and that's exactly what netlink what neck you does I'm listening to things like routes MAC addresses IP addresses lldp neighbors bands sensors interface counters you name it and I can run this on any device that runs Linux so it's supported on both cumulus linux ubuntu debian and Red Hat and now that I have all this data about my environment I can start to ask it questions and so I can ask questions like if I'm anywhere in my environment is BGP working as I expected and Nikki will look at all those devices and all that information it'll say you know I looked at those two hundred and sixty sixty device or sessions across your 42 nodes and everything is working I can hold data or information from any device from any device so you'll notice I'm on leaf 30 I'm gonna cheat here and I'm gonna look at host:oh - I can see into the Putin - box running 16:04 who is host OH - connected - I see that it's connected to leaf - on Ethernet one what are its IP neighbors what are its ARP entries I see an ARP entry here for 10 1 2 . 11 all this is about host Oh - even though I'm somewhere else in the network is if we think about our flow as network administrator's what we end up doing is logging the one device running show MAC address table login to the next device if we're advanced we're using team ox you know if you're if you're leet you're less leet you've got like 30 tabs open mm-hmm or if you're like me you just get like a really tiny resolution and open like 400 putty windows what a man I'm back I actually just hate myself so much that I start a Windows VM yeah and then run putty in the Windows VM I believe punish myself I believe you would do it I really do so I can look at a MAC address for a neighbor right and I see that this ties to the 10-1 2.11 host this 44 ending in 39 and I can even say who who owns that maybe I don't know the device it's connected to and so I can say Nick you who owns that MAC address where do you see it in the environment and nephew tells me well he see that MAC address in VLAN 12 on leaf 2 and on leaf 1 leaf - learned it through vni leaf 1 learned it from host 1 and we can do things like well what would you do if you received that MAC address you received a frame destined for it so I can do a trace route for l2 or l3 where that MAC address from any point in the network so again I'm on leaf 30 leave - what would you do if you got that MAC address leave - is gonna look at that information and it's gonna say well it's over a VX LAN tunnel so I'm going to encapsulate it into the NI 12 I can then ecmp out my four links slops one two three and four we call our front panel ports swp or switch ports that'll land on spine one two three or four port one that'll egress to lethal onesie and I and you know what all of those will end up on host o1 but this is great everything's working what if I break things like the size of my terminal window so I wrote using net cue we're gonna go a little bit more into what I can ask net cue and how do I access that information but I used NICU to build a chaos monkey so if you're familiar with Netflix they have this thing called chaos monkey and what they call the simian army which is a collection of tools that goes in and breaks things randomly to make sure that it's resilient know as network engineers that is our job I put two devices in and parallel or install two supervisors because I hate money and want redundancy there's lots of reasons we do things but we're all about that resiliency and then much like a backup solution we never tested and so Netflix had this idea of chaos engineering where no no no we are definitely going to test it we're gonna go in and break things and for us the challenge at breaking things is one organizational but two it's hard to do right I mean we are we are kind of in the sylvie infancy of automation where we're still crawling to push configurations how do I automate or randomize chaos in my network besides hire interns so I can use net queue to pull the all of the devices in the network figure out the services they're running and then break some of them and I call it the chaos tamarin a tamarin is a tiny little monkey that weighs like one or two pounds because I have a little chaos monkey and so I'm going to break bgp on three devices and so it's gonna find all the devices running PGP in the environment using net queue it's then gonna log into those devices using SSH not using that queue and it's gonna shut BGP off on them and so now I have these three broken devices that's great I put a print statement in there so I can see that but what if it was my actual production Network well I can see exactly what happened I can see I have six sessions down because I broke three in the remote three sides are gonna break as well and so I think this really just kind of shows one the ability to look at Nick you but - what is it showing off what is it what can I see with it you know going going beyond this something that's cool is Nick Q is built on again simple tools so instead of building something brand new and unique and different and interesting we built something known and unique and interesting so we took the Redis key value store a distributed database application that's part of what net Q is so every switch in my network is now part of this distributed database and they're pushing their data into that database so it's just a database and I ask it questions you absolutely can so we can do things like well what are the tables in that database so when I run net küçük BGP I'm actually querying the database pulling data and then we're providing some intelligence to actually process what comes back from that that sequel query if we will but I could build my own queries so we give you the things that we think everybody cares about but you can build the things that you care about so let's say I want to look at all the devices in my network and I want to sort them by uptime so I can run a query where I'm going to select the hostname in the SIS uptime from the nodes table and sort it by cyst uptime and you can see that four hours ago which was you know for a really fun adventure after this you could look at my committee history to see the panicked building of this immediately before we got started like it's the duck mode where it's really calm on the surface and they're just kicking really hard underneath but I can this is great I can look it up time whatever but what if I want to do something like see how long it takes a route to get through my environment so let's uh let's look at a loopback address on this device the 1000 30 I P I'm gonna shut down that loopback interface and make it get withdrawn wait just a second bring it back up so I just create a route flat and I want to see how long it takes for that environment to propagate that route flap and reprogram so let's look at I'm gonna query from the routing table where the prefix matches what I'm looking for and sorted by time because everything that we record we throw a timestamp on it so I can see 17 seconds ago it showed up on the 30 because that's where it started 12 seconds ago spine1 got it those are my immediate four peers they're gonna be the first ones to receive that route after that I'm gonna propagate to the rest of my network so I can see that twelve point seven seconds to go my whole network converge for that prefix based on those queries one of the other things I can do is because I have timestamps on this well let's go back in time so again let me look at this trace route or VX lan that we looked at earlier so from leaf to everything's great everything works so just to chat when you run the trace route it's not actually sending anything that is it is analyzing the control point state of the path right so it's it's theorizing what would happen based on state that it might it's exactly right okay so I shut down one of the up links I'll run the exact same trace command now I only have three paths that's expected but this looks good NICU doesn't know if this is broken or not so what did it look like so you know that issue or they're like hey yesterday at 4 o'clock it didn't work but then everything was fine by 405 why and you go you know sis logs or gtfo and then I never had logs well now I can look back in time and I can say huh that's weird something changed how long does it store that data for based on the memory of the server so we have a server called the telemetry server well that's sucking all this up and running the queries it's only based on memory and disk of that telemetry server so it forever it uses what you give it infinite until the hardware runs out we're using we're using advanced blockchain technology queryable via an API so today not directly meaning that it's just Redis so you could clear it through a Redis API but you can clear it through the neck you agent API there is no external there's no like restful api on top of it today sorry did you say that all this is running on a separate telemetry server it is the agent runs on every device right that agent is part of a cluster which is all the agents plus another VM running running thing so it's not it's that other VM I was checking okay good just making sure I understood and so I can look back in time and say well what changed in the last five minutes on that leaf and it tells me well you lost the BGP pr1 I saw him too you change because it went down you lost some ipv6 routes and a whole bunch of routing changed as a result of ecmp changing so again I can go back in time and see exactly what happened and I can even take this tool and provide it to my Systems Administrator and say don't call me and let them query this and they can run this agent on their devices and see those things and I've shown some Network focus things but we've also added integration for docker wait so Nick you is an agent that you can run on any Linux system it doesn't have to be a networking note at all that's exactly right query this database and tell you the state of your network yeah so if I look at you just blew past that one Pete like yeah you gotta give us a minute to absorb this that's on me absolutely this banking no if I look at this device it's in a boon to server called host o1 Thank You host o1 show IP addresses right so I can look at the information from that host all the hosts in this environment are Linux devices so this one's running docker so I send our interfaces side to it that I can see and then there's a bunch of other fields that I can query about those hosts but we're listening the docker swarm so I can start to do things like fill me the summary so I can see the nodes that are part of that docker swarm I can see how many networks what they're how many containers they're running things like that if I look for the services the only container service I see in my environment is this thing called Apache web I see that it's running on four devices well which devices is it running on what's their connectivity in the network and I can see it's running on hosts three four five and six who connect up to Leafs three four five and six so I'm starting to blur that line between the network in that container land and as network engineers we absolutely know what's going to happen is these super ephemeral containers are gonna pop up and down and it shot like you know rats in a basement and they're gonna go why to the network break it and you're gonna go it didn't that's what your what the container thing does the feature of containers so I can even say well what containers are adjacent to the interface of a device so I'll look on leaf five show me the containers that are adjacent on port five and it'll say well on port five I've got a patchy web in cumulus row so all of these devices running docker are also running routing on the host BG be peering directly with my top-of-rack switch so is it EGP tab-completion for everything not running BGP there but our net queue on that one if I look on the leaf it's attached to I actually see that I have a BGP peer to that top-of-rack switch so let's look at things like impact so what would happen if I needed to do a software upgrade on me fo5 I have to take it out of service I don't know what's running underneath of that there the container servers well I can actually see that it's part of this Apache web cluster and there's three other replicas in here but you're gonna kill one of them it's single home it lost its dual uplink it was single home from the beginning whatever that is I as the network engineer now and know exactly what I'm gonna break who's gonna call me and just like everything else I'm gonna record the changes so if I jump on a device and let's say spin up a simple container called hello world which is already running there so I can't do that so I'm gonna span up this new container alright it doesn't matter what it does but I'm gonna start a container it's gonna live for one second print hello world and then die well I can see that again from Nick you and I look back in time and I say well you flap the router on the host container a couple of times you stood up this Apache web thing and then you just launched and deleted the hello world application and so again I start to get this visibility to know what happened and so again when I get that phone call that says hey why did you break my computer environment you go when you talk about man you your your containers just turn themselves off that's not on me not the networks fault so that's a stupid question now it's good good question is this the net queue is that the nco you piece you're not having to go into another CLI and then back into Linux NICU has its own CLI but it's actually built again on those same common building blocks so knit and CLU so I can do an IP linked show and get this Linux the output or I can do a net show interfaces with tab completion and get something that looks signaling less awful so that's a CLI application running on it on Linux and net queues CLI is actually that same CLI library so it is a separate application but it's importing its own and seal you one of the things I think is really cool I was it net so net help told me that what I'm actually looking for is net example you know what I mean like you're not having to go into a different CLI to do certain things in fact that's right it all it all happens from from the the bash prompt gives me that tab completion in those game and until you itself has built-in help so I can say well if I wanted to build OSPF unnumbered between these two devices called spinal one and leaf o1 given this information these are the set of commands that it would run and this is how it would verify it we look at something more complicated because you're not even going in to a see a lot you're invoking a CLI from the bash prompt exactly the CLI is just an application that's running on top of Linux that provides a different interface into all the same information right that's exactly right we've also been seeing a lot of things like orchestration and whatnot this week are you involved in any of that stuff as well yeah yeah absolutely so what we've done for orchestration is doing absolutely nothing but being Linux having every Linux to Linnaean on the planet work with us so I'll give you that exactly a concrete example so it's not it's not a box orchestration tool that we've been looking at that is correct so it does rely in this case on a little bit of skill but that's not necessarily a bad thing you know just kind of piggybacking off of what it requires in skill it makes up for in extreme flexibility compatibility absolutely robust robustness no but very good so good robustness is so I'll just use hands I use danceable to build this and really fair peak if I could throw it out there ansible has a very low starting kind of barrier it's very very quickly it's not a difficult thing so it's not like it's full of emotions yes which is perfect for this and that's the thing what's beautiful is because it's all virtual I can build a virtual lab I can then use ansible running virtually and then when I break everything I can just kill it and restart it the whole via and the whole lab whatever you know so I can do something like run an anthem we'll play book that's invoking standard ansible libraries not the cumulus ones but the off-the-shelf Linux ones that are being run on tens or hundreds of thousands of Linux devices every day and reprovision and fix my environment if I can take that thread like it's just a little bit further what are the things I think we've heard from another number of folks is this idea of kind of becoming the platform and what I think I hear you're saying is that that platform is already here it's Linux mm-hmm and then but then so then taking that step further is if you're gonna be running you know in an enterprise and you may have folks who are admins or operators versus architects or engineers maybe there's more setup involved here right so the tool and the fancy tooling we were just referring to makes it a lot easier to operate but you could set all that up ahead of time so maybe they just think those may be a little bit higher bar to entry did you agree that you know there's a there's a great there's a great xkcd comic talking about like building a script versus time right and so as a network engineer your time is effectively literally growing forever on the amount of work you have to do do this stuff you have a really high barrier entry for that first day and then you just like go home at 11:30 in the morning every day for the rest of your career and so there is that setup piece but again this is why you know we haven't taken a dogmatic approach we built a CLI we support tack acts you know we we still do l2 even though we look at l3 is the world so we have we say look this is the way enterprises should do it here's the place you can start with what you have today and you know let's help you get there on that journey

Channel: Tech Field Day

Views: 5,703

Keywords: Tech Field Day, TFD, Networking Field Day, NFD, Networking Field Day 17, NFD17, Cumulus Networks, Pete Lumbis

Id: Eqq2CH9MX6I

Length: 38min 15sec (2295 seconds)

Published: Wed Jan 31 2018