Nornir: Solve Big Problems Fast, Interop 2020

[Music] hi everybody my name is brett lykins i'm a managing consultant at network to code and today we're going to be talking about nornier and how you can solve big problems fast with this tool so first we'll cover what is nor near you know just the basics of what it is when you might use it then we'll talk about where it might slot against some different tools that are available in the marketplace today and then we'll really dive into the internals of it some of the basic concepts that you need to know in order to start utilizing this tool and then we'll take a run through a couple different example use cases for when you might reach a new toolbox and just pull that out so first what is nor nir so nornier is a python based automation framework it uses an inventory of hosts on your network that you give to this tool and then it executes tasks against those hosts it's extendable very easily it's built on a very nice plug-in architecture and it's agentless there's nothing to install on your network equipment or anything like that and the github link is here in the slides as well so fairly recent project started in december 2017 and uh we're up to a 2.0 release 2.5 actually right now 35 folks who've contributed and it's up to 644 stars on github which is pretty nice so go give it a star if you've got the time and um it's definitely growing uh 3.0 releases around the corner and we'll talk a little bit about that coming up here in the slides so one of the questions that comes up very frequently at this point is what about ansible is is an ansible a python automation framework and and to that i'll say first nor near is not ansible so the main difference for me is that nor near has no domain specific language you don't have to learn how to write specific nor near architecture in the yaml files the way you would with ansible with building a playbook and building your plays and the real key benefit is it's a 100 python there is no need to go into different tools to do troubleshooting or anything like that you have a python ide you can troubleshoot your problems that you have with your nor near automation you have access to all the native debugging tools inside your regular python ide now with that said ansible certainly has a place ansible has a little bit lower barrier to entry in the fact that there are a wide array of community driven modules and playbooks around a lot of enterprises have wide usage ansible inside their organization already your server teams may be managing your servers with a configuration with ansible so you may be able to build on some of that with network automation uh one of the other key things to note is there's the functionality of awx or tower interval tower to allow you to orchestrate or execute your playbooks in a very rigid and structured fashion and you know nor near does not have that it's very not batteries included in the same way it has a lot of power but it doesn't come with some of the things that allow you to quickly and easily bootstrap in some of those same ways but we'll we'll cover a little bit of that further on now again there's a plethora of automation tools out there and people usually bring up you know what about salt what about this what about puppet and so what i'll say to that is nornier was originally started focused on automating network devices and you know the founders of nornier came out of people who kirk buyers makes netmeco people who made napalm network focused automation tools in python building this framework and was really to fill in some gaps in the tool sets ansible traditionally was not that great at network device support it's certainly gotten better and you certainly use it to great effect nowadays uh salt though and some of the others require proxy agents or physical agents on devices which were really non-starters so there's some key pros and cons nor near allows you access to those native python tools you can really easily extend its capabilities from the front or back side because it's all python and it has first class support for your network devices so it's built around the really robust network automation community that's grown up that uses python now again some of the cons it is a newer tool there is a community but it's not the same size as ansible by any means but it's growing and it has less of that batteries included you do have to have some python skills to be able to bootstrap up the things around it to equate the same functionality that you would have from ansible now why you may want to go tackle some of those cons is if you already have a python framework in place where you're doing some tooling in your network with python it's really easy to integrate nor near into that tool suite and expose something via flask api for example for to allow somebody else to run a task on your network devices or for you to automate against an api instead of directly against the network devices and one of the things we've seen a lot of and i'll cover in a use case a lot is it actually integrates well with nutbox which is a free open source uh dcim tool that has a lot of uptake in the network community right now and uh you're able to integrate warner directly with uh in netbox as a plug-in so some really nice benefits there and while while speed isn't everything nor near is fast and just like i said here shout out to patrick ogstad it is considerably faster than ansible at equivalent network tasks now speed isn't everything like i said there because there's certainly reasons that you don't care about how fast it runs your backup as long as it completes within 24 hours say it can take as long as it wants to run backups or you know it's the idea of eventual consistency is important whereby if the job gets done within a timely manner how fast it gets done may not matter it depends on the job but don't necessarily go chasing speed at the cost of all other things so let's talk about the internals of mornier a little bit as i said before it allows you to take an inventory that's comprised of hosts groups and defaults and then it allows you to execute tasks against those hosts which tasks are just python functions uh with some extra things wrapped around it that make ignoring your function but under the hood they're essentially just python functions now the inventory itself as i said earlier it's a highly extensible architecture it's built around plugins uh some default ones that you see a lot of or simple inventory plug-in which is just yaml files to keep your inventory in ansible integrations for plugins nsot uh that restores the truth tool and again netbox uh which is one we see a lot of use of so you just you keep your inventory in these other systems or keep it in the yaml files and nornier can integrate with that via its plugin architecture you have not one of those you want to integrate with something else you have a custom cmdb you're using something ingrown in your house write your own the plug-in api is very easy to use and uh linked here in the documentation to how to actually create that but on the near read the doc site there's a very good description of how to write your own inventory plug-in now most common especially for starting out and for for simple examples you'll see simple inventory it's just a yaml file structure where you have hosts in a file and you have a group settings in a file where you can apply attributes to those hosts based on group membership and here's a simple example of a group's uh sorry the simple inventory plug-in here is example of a host file here so you have your simple description of your your device hostname for an identifier you have the hostname how to reach it password what group it's a member of and any other arbitrary data you care to store about that device maybe you want a location or something like that you can store it in there in the groups file it's very similar you have the group that you called out earlier and any other attributes that you wish to override so i want to say that these are cisco asas here's a username um any other arbitrary data you wish to store for all of the asas in that group so one of the other functions the structure is to allow you to keep it really flat really simple structure i would not use the simple inventory plug-in for more than about a thousand two thousand devices because it really starts to get very cumbersome that's where i would look to keeping your devices some other source of truth like netbox and integrating with one of those plugins but for our example we're going to use something simple here so there's just really in this example i have two asavs in my my simple inventory now you have your inventory now you need to set up tasks to run against those hosts in your inventory again nor your tasks are just plugins just functions and all that the requirement around turning a python function into an ordinary task is that it needs to accept a nor near task argument and order task object is one of the arguments and i have an example here it's a simple task a task i'm going to print that i'm doing things to this task.host.name and what you get by bringing in that task object is you get access to all the nor near internals so when this runs it'll just run that i'm doing stuff to asa v1 it will print out the host name there so you can actually take tasks they'd be really simple functions or you can group them inside of one another you can have tasks nested with subtasks and subtasks under that you can also create your own there's some baked in and i call here there's some some pre-defined ones that come with nor near local file operations simple network command execution for anything more complicated you're usually going to want to write your own just to put a put your logic into that task itself manage how the results come out things like that but you can certainly just use the canned ones for something simple so um here these built-in tasks they're nornier.plugins. and again i'll talk about this a little later some of this is changing in warner 3. um the plugins will be not built in the same way but the the idea still stands they'll just be accessed a little bit differently we'll cover that later in slide deck so one of the things to call out here you have options with netmeco or napalm which are very commonly used very well-known network automation tools and you have plugins that are able to be used in nor near to use those automation tools so you can send a command to a device with net nico you can use uh napalm git to get facts about the device things like that so you have your inventory and you have your hosts what do you need to do to do things with it so you need to run an initialization you need to start up the nornier process so that's normally done via nornir.initnornir is the function call and it basically takes your inventory reads it in reads in any configuration settings which we'll talk about in a moment here and prepares everything to run uh execute tasks on your hosts now it's a very flexible configuration as a call here there is some ammo it's not enamel free but uh it can be almost for configuration not for your logic like ansible but uh you can either have a configuration yaml file or just pass parameters to an ordinary or a combination of the two uh some of the example things you want to control is how many worker threads are you running what inventory plug-in are you using so it knows where to get the inventory those kind of things logging settings things like that here's an example configuration file you don't need to read the whole thing this is just a sample of it can be very simple and straightforward i could say how many threads i'm going to use where i want to send a log file what what processes i want to log from or what libraries in the in the tool i want to log from and then what inventory i'm using and then some information about my ssh configuration so that my host is running this knows how to ssh to those devices if it needs to use a specific key combination or what have you we've got the configuration file and then we call in an ordinary and we pass the configuration file to it so in initinornier and config file equals wherever i've stored that yaml file and so now i have this gives me an instantiated nor near object which i can use to run tasks against the hosts that were in this inventory file that i referenced in config.yml so the inventory file backwards here the inventory file was listed here and that's where i thought where the hosts and groups are so now i can just feed in that config file and away we go now you can also pass parameters so for example say in this testing i wanted to run only 50 workers instead of 100. so i can override the config file settings with a parameter that's passed directly to it and just run 50 workers most settings are accessible the same way you don't have to use a config file if you don't want to you can also just pass parameters it's up to you how you want to use it usually i recommend storing it in a config file so that the config is checked into source control it's in your repository and then you can override that in testing if you need to override anything specific with settings the parameters now once you have that instantiated nor near object so we have minor here you can call dot filter on it and actually filter your inventory because not all the time you want to run everything on all the hosts in your network so say i only want to do it to cisco devices i have some cisco some juniper some some arista i only want to run it on cisco i can say where vendor equals cisco and the device is not colo because that's some arbitrary information i've set on these devices whether they're colo or their managed devices i can run that that way and so i can pass those as keyword arguments to dot filter and it goes through my devices in the inventory and returns only devices that match this where vendor equals cisco and iscolo is false and so i will get a filtered subset of inventory called filtered hosts and then i would run my tasks against those filtered hosts you have a couple other filtering options you can get really in-depth you can create your own python function to execute filtering you can just call a filter func instead so i made a simple function here that returns anything with the device id is in the arguments i provided to this command line tool it will return me those hosts and lets me filter it that way you can really get as complicated or simple as you want with filtering like i said you can call this filters one of the other examples they have if you're familiar with the django framework the concept of using the f object to filter your inventory as well i don't go into that in detail here because it takes its own amount of time to sort of go through but i recommend checking it out if you really want to dig into the filtering their docs are really complete really good around how to use f filtering so i recommend checking it out we have our tasks we have our hosts we have filtered our inventory potentially now what do we do so how do we execute the tasks on that host well it's dot run so you have your nor near object your filtered hosts or just your instantiated inventory you do dot run on that object and pass it a task name so i'm gonna do dot run whatever my task was that i wanted to run and it's gonna run on the full inventory if i've instantiated the full inventory a filtered subset if i've run a filter and what it returns is a results object for each of those hosts which is a dick like object with attributes about the job that ran the pre-can tasks pass back some things you can make your own tasks and define what's in that result stick but it just is essentially a way for you to manage and corral the results of that task and then access them later for reporting for generating a view on what happened when your task ran things like that so here we have an example execution so one of the things that that i see here is convincing all of the things that we've covered so far so i'm importing in it nor near from the uh main nornier uh install installation there i am creating a nori object by feeding in a config file i'm defining a task which all i'm doing is the task we saw earlier doing stuff to task.host.name and then i'm going to run that task so when i run this so this is just straight copy and paste from an interactive python shell it prints out doing stuff the lab host.test so that was what that host's name was in my inventory so now i have instantiated nor near i've created a task and i've run that task against the host in my inventory in this case the task was just printing something it could have been ssh into the device and running a command it could have been hitting an api and pulling back results about some device or doing something on the device's behalf you know the world was wide open to you which is sort of that power of more near um but definitely it's very simple in just the building blocks to get you going you just have to put them one on top of the other so and again it is requires some python of knowledge i'm not discounting that part but that's where if you have some of that python knowledge this can really come into play so let's talk a little bit about what we've covered so far and with nornir you get an inventory of hosts you have host file if you're using simple inventory you have a host file and a groups file and potentially default where you list in the groups a default attributes to apply to all your hosts if you wish to set it that way it's basically a parent group for that purpose of structure you have tasks that you want to execute on those hosts and there are many that come pre-canned or pre-built by the community and you create a configuration file which would be the recommended way by me to to set this up or you can pass attributes to init or near but the key is using it in order to build that nor near object that represents your inventory for you to run tasks on so you instant you you initialize that near instance and then you do dot run on there with a series of tasks so you could do dot run once pull your results you have the same inventory already ready do dot run again to run a second task you know how you want to structure that is really up to you there's a couple of different ways to go about it one of the ones i would recommend is just starting out with sequential task ordering is definitely better than trying to work with nested tasks initially because sometimes sorting through nested returned results can get a little bit complicated so i start small one task one result and try to work with that to get a feel for how it works when you're working with this but definitely you can do dot run to call tasks which call subtasks and on down the tree so to speak it's up to you how deep you want to go there but you have that capability but the main thing is you have an ordinary object you do not run so we talked about what norner is some of the you know alternatives in the market as well as some of the internals here now i really want to dive and spend some more time on use cases and that's really where sort of the rubber meets the road in this so when we have our different use cases some of these are just a couple of quick examples um we kind of dive into these i'm going to leave time at the end here for questions absolutely i'll be answering live while we're going over this as well so any questions that come up uh but this is where i really want to have a conversation about some of the different ways in which you can use it put those ideas in your head and sort of look at when you can pull out in order to use your your examples and your own inventories and things so for the first example we have a large company a technology company that bought another company and company x has a very robust network automation platform they have tens of thousands of network devices out there they have a well-established api base running in golang some some running in python with flask and they're exposing web services out to their end users and it's very it's it's the it's the panacea it's everything you want out of your network automation company y is not so much they're not as far along on your network automation journey and that's just where they are and that's fine but a lot of things that you run into when you're in that integration and that that merger acquisition scenario is your existing tooling is not compatible with the new tooling from company y you know they may have a different cmdb they may have a different inventory system different authentication systems they're pointing to different tacx servers or they have no tacacs who knows and so you really have to step back and say well i built all these tooling how do i use it on this new acquisition and you don't have a lot of time to do that because your auditors your security auditors your compliance auditors the leadership from company x they want to see the same kind of reporting and data about the company-wide network that they're used to seeing about company x and so in this example you know you're on a tight deadline you don't have a lot of money to spend for uh going out and building new resources and new tooling it's just got to happen so this is where i was able to utilize nor near to take the company-wide inventory out of their cmdb and build it in a simple inventory yaml file because it was around two or three thousand devices i think the one we were working with there i was able to utilize the existing python applications and flask apis that were built inside companyx's network and build some just add onto them with some route views that were specific to company-wide resources but on the back end they call gournier tasks using that simple inventory that was quick easy way and we were able to do that in just under around a week for the first like okay we need to tackle this problem we have access and let's turn out a prototype that was about a week's worth of time because that was a shop that was experienced python developers they were able to take this and say i can't extend my own tooling yet but i can use dornier to quickly access those devices and so that was sort of where we were just strapping to the existing web frameworks that existed because they were built on python and used nor near to talk to those devices to manage task execution on all those thousands of devices and so very good turnaround on the time that was spent on that into you know getting your results there another example that we've run into is say we've got another fictional company company d they've installed the netbox free open source dcm tool and they've got all the network devices in there and they kind of stopped and said well now what do we do with it and one of the things i've seen is you know people have that inventory they have that source of truth it's use it for automation and so one of the things they ran into and they wanted to do something quickly is they they had a six-week lead time on getting additional vms spun up or a load balancer and firewall config put through and they had all these these hurdles in front of them and they weren't really as deep a python shop as the example in company x so what they were able to do is to take the net box vms that they have and all that associated infrastructure with the load balancer and all that deploy onto the existing django app that is netbox a view that used the uh used nor near to execute jobs against the different hosts in their inventory uh they use django rq on the back end to queue up job processing and it really helped them get going and bootstrap and as it says here they went from their poc to production in about a month or so it was really good use of using nor near with an existing tool they already had netbox to go and it's gonna be even easier nowadays with the netbox plug-in architecture because this actually predates the the plug-in capability that netbox has today so definitely good use case for an ordinary if you already have some of these resources so in short you know you want to know how can i get it it's on github you can just do pip install in order it's on pi pi and where can you learn more their docs site is very robust you can go to mourinho.readthedocs.io great documentation there's a slack channel on the network code slack instance uh hashtag mourinho which is very active the developers are in there dave veroso and all those folks are in there and there's also a discourse group because slack is a bit ephemeral if you want some long-term record of questions and answers i recommend checking out this discourse group well i will say nornier 3.0 is is imminent should be potentially by the time you view this it will be released there's two key uh feature additions one is the separation of plugins out of the built-in to being in their own independent repositories with which allows you to only install the plugins you care about you're not going to use napalm you don't need to install the napalm plugin and allow you to just install nornier then any nornier associated plug-ins you care about which is very powerful way to break apart and use that structure it also changes up how some of the task execution runs and lets you choose different runners for that i'm not going to dive into it too detail here because some of that's still being baked but just a heads up that know three is on the horizon but it shouldn't be too difficult to change your number two to three but something to be aware of if you're starting a project today so in summary it's a python automation framework that allows you to use with your other toolsets it's very fast very extensible it uses network devices as first class citizens and there's no need for agents or anything like that and there's common plugins for most of your network device tasks and there's an active community although small this time still growing uh definitely recommend checking out if you have some python schools or if you don't there's help needed with documentation all kinds of things i recommend getting involved and so i want to say thank you thank you for taking the time to listen to me today and if you have any questions feel free to reach out to me on twitter i'm on github like speedy or you can reach out to me on the network code slack channel i'm at brett there so thank you very much for the time i hope you all have a good day enjoy the rest of the conference you

Channel: Network to Code

Views: 668

Keywords:

Id: l7whWE4oi_Q

Length: 25min 18sec (1518 seconds)

Published: Mon Jan 04 2021