Creating pyATS | Genie Parsers and Unicon Connection Plugins

what's up everybody coming to you in a slightly different format because i'm a little pinched for time but i wanted to make sure i got this video out there for you all to see one of the projects that i've been working on recently is i've been working on automating a dell power switch device running operating system 6 dell os6 and what i wanted to do was use pi ats in order to actually run parsers let me show you real quick if i bring up the terminal from my dell os 6 appliance one of the things you'll see is that the dell os6 it doesn't really support things like i already got to type my password here we go there we go one of the things is is dell os6 appliances are cli based appliances they don't have an api they don't have netcomp for so anything like that so this is where pi ats could really come into play and help us develop automating solutions for platforms like dell's power switch appliances we can build parsers that actually allow us to parse this output and turn it into structured data and that's what i've been doing so let me introduce you real quick to what is going on here and how you can get started working on this too there are a couple ways that we need to actually use pi ats to actually perform parsing output the first one is we have to build the actual connection uh to the dell os6 the actual driver that logs in and can figure out where in the cli it is it can actually look at say the command line let me bring it back up and it can see whether or not it's in the user mode or can see if it's in privilege mode or it can see if it's in configuration mode that's you have to actually build the mechanism that can connect in and figure out where it is uh and navigate between tiers and actually issue commands and this is what pi ats calls the unicon this is a plugable solution so that way you can develop your own connectors and uh just you know up front like you have to have pretty solid uh python skills in order to do this so if i'm on github.com cisco test automation here we are github.com cisco test automation i can go into unicon dot plugins and this is where you'll be able to actually build the connector plug-in which ssh is into devices and uh can navigate through the cli and execute commands the good news is is that most of the heavy lifting is already done so actually let me just show you real quick from the basic repo here i can click on source unicon plugins and you see the list of devices that are supported um for the most part you see like things like ios ios xc ios xr all of these inherit from these this generic folder and all of these classes and functions that are in the generic folder so most of the heavy lifting is actually handled by you and you just end up tweaking your connector plug-in to work with that and i'm going to talk about that as we get into the actual code itself jumping back to cisco test automation once you actually have the unicon plug-in in place and you can connect into your device then it's time to actually create the parsers themselves so from cisco test automation i keep jumping ahead there's the genie parser section here and this is where the actual parsers are written so here you see source or src slash genie you'll give that a click you'll go into libs and then parser and then you see the list of parser appliances that are here so we can take a look at something like ios and you see each of the commands that they've built the parsers for right let's pick on something like show rip and actually that's it's pretty basic let's go back and pick on something a little more uh in depth how about show interface that's going to be a good one yeah here we go lots of classes here um in fact it looks like it actually inherits its configuration from show interfaces ios xe so it's another bad example we'll find a good one here i promise um shows spanning tree nope still inheriting okay well we'll get to it don't worry about it parsers are this is where this comes in so when you want to build your own plugin or your own parser this is this is going to be your starting point is github.com cisco test automation so the way this works is you basically clone these repositories to your own github account and then you can make changes on your own account and then you ask them to pull your changes in and that's what i've done with unicon at least so let's take unicorn for example we go into unicon and then up here in the top right example you see a fork button when you click that it takes a copy of it from cisco test automation and puts it on your own account you have forked the code and now you can make changes to it on your own account and when you're done you create a pull request to ask them to pull your changes back into the main so if we actually go to my account real quick i'll go to github.com data knox i'll go into my repositories you see i've got unicon plugins and jeannie parser already here it's because i forked it and it says right here in the little text uh forked from let me hit that because that was a link uh it says forked from cisco test automation unicom plugins and forked from cisco test automation genie parser so when i go into unicom plugins now and i go into source unicom plugins now we see my dell os6 folder here and the scripts and code that i wrote to make the unicom plugin for dell os6 if you click on it you can see this is where i wrote it i put my little author there section largely inspired by the unicon plug-in example so uh now's where we're going to actually get into the we've talked about what we're going to do where i was creating the dell uh os6 unicom plug-in as well as parsers now we're actually going to get into how you can do it too first things first i use the pi ats docker container for my development if you watch other people's guides or read through documentation they'll tell you to use a virtual environment i absolutely could not get that to work on both windows or wsl um inexplicable why so i just did it all in a docker container so without further ado i'll bring up a terminal and we will say we're actually just going to copy this right here this docker pull command right there we'll command copied windows terminal and pull it so it's going to download this docker container finds out that a lot of it already exists which is pretty cool this is going to get the latest version of it and there we go so it is now pulled so here's what i am going to do we're going to say docker run i'm going to say i want it detached and interactive that way vs code can connect to it and we can do our automation and work with it there we'll say i'm going to give it a name we'll call it like pi ats dev and then we will run this container here going to copy it and paste it cisco test automation pi ats late and i'm going to tell it enter in the bash mode so i press enter there and now the container is up and running so we got a little bit of work to do now to set up this environment at this point once you've run the docker container you want to go to github and fork either the genie parsers or unicon or both whatever you want to develop on if you want to work on only parsers and you don't want to work on connections you don't need to work you don't need to fork unicon unicon is only when you want to create a new connector into a new platform uh if you only want to work on parsers you only need to fork the cisco test automation genie parsers repo so that's what i've done i've forked both of them here so now that my container is up and running i'm going to fire up vs code i'm going to click in the bottom left corner and i'm going to choose attach to a running container and there's pi ats dev right there so this fires up a new instance of vs code and in the bottom left corner i see i'm attached to it what i'm going to do is i'm going to open up my terminal and give it a print working directory it has me in root so i'm going to cd out of that into the the actual root folder and over here on the right hand side i'm going to choose to open a folder and actually i'm going to delete this out here i'm not going to do root see i've got just forward slash at this point i'm going to go into pi ats and click ok so over here on the right hand side i see pi ats fires up and running this is the actual pi ats code that is going to execute the actual like raw pi ats work so in terminal i'm going to fire up a new terminal again i now see i'm in pi ats uh here's the thing pi ats in this container runs in a virtual environment so the first thing i need to do is go source bin activate and now i'm in the pi ats virtual environment which actually has pi ats running in it so the idea now is that we are going to be able to pull down our repos that we forked and when we make changes to it we can override the pi ats environment so here's the next thing this pi ats container is running the container image of python 3.6 lite so that means it doesn't have git installed and it doesn't have make installed so we need to get that so let's give it an apt update there we go now i'm going to say apt install git and i'm going to say yes to install git i'm also going to say apt install make m-a-k-e there we go so now we have both of these now in this pi ats folder i'm going to make a directory uh i'm just going to call it working so it's going to be my working folder now i'm going to change into working and here's where i'm going to clone the two repos that i created earlier i'm going to jump back to github real quick i'm first going to copy the genie parsers so i copy the url right here i jump back to vs code and i say git clone and there we go it's cloning the genie parsers right now and there we go now i'm going to clone the unicon repo which is on my side and clone this again this is my repository not cisco's repository after i have port their code so there we go now it's going to clone unicon so if i give it an ls i see i've got the genie parser and unicom plugins and now over here i see my working folder i see the genie parsers and i see unicom plugins let's start with unicom plugins for this i'm actually going to close the terminal and we're going to navigate from unicom plugins into source and then into dell os6 and we need to talk about these uh in a lot of detail this is honestly the hardest part of the whole thing if you're making a unicom plug-in this is pretty challenging remember that the big thing that you're doing for the most part is you're going to be inheriting from other plugins particularly the generic plugin also because dell os6 is so incredibly similar to ios i in some cases inherited directly from the ios v plugins so let's start with the init file the init file is mandatory all of these files are mandatory i'm about to show you uh and it is mandatory and this is what initializes the plugin now here you can see i am inheriting from the ios v single rp connection basically a single routing plane or um not a dual head uh connection not like a master chassis or anything like that um this for the most part there's not a whole lot to it i just identify the os by name os6 and i import a bunch of stuff from the other files so my recommendation to you is when you create your new folder for unicon create all of the files even if they're empty create init patterns services settings state machine and statements uh and then honestly my my recommendation to you is just copy and paste like my code here in each of these files and then edit them as needed so the init file is what initializes this uh this class um or these classes and the connection itself there's really not much to it we're importing the state machine which we're going to talk about we import the service list which we're going to talk about we import settings which we're going to talk about the rest of it is basically inherited from how ios v is imported patterns this is where we set up how it is able to determine where it is on the cli it's able to determine if we're being prompted for a login it's able to determine if we're in user mode it's able to determine if we're in privilege mode it's able to determine if we're in in a config mode and that's like config config interface config vlan beyond that it's able to determine i had to add this one in here so i had to edit a lot of these and i also had to add password in here because when you may have noticed this already when you go into privilege mode right here i typed enable it prompts right here for password colon that's what comes up on the field and this is why dell os6 is one of the reasons why i had to create a new unicom plugin so this just sets up the actual regex expressions for identifying these patterns uh other files here are what uses them services i don't really understand what's going on here full disclosure uh i know that this is how it tells the dell os device uh to execute commands for the most part i copied and pasted this full disclosure uh there's just not a whole lot uh i you can see actually right here like in the in the top i said contents largely inspired by the sample unicorn repo they do have uh a sample unicon repo where you can see like how to build your own plugins it has a lot more comments on the text on how to do it really for the most part here where it said things like ios v you can see i i'm again inheriting a lot from ios v down here the very bottom basically where it said ios in the functions themselves so right here on line 39 i changed this to be return dell os x 3 and then on the very last line right here i identified that it is initiating the dell os service that's all i got for you on this one i know this is an important file uh but i just there wasn't a lot for me to edit and there you have it settings this is an important file this is where when it gets connected and ssh is in under the hood when an ssh is in it enters a series of commands so in this case in my case it enters terminal length zero if you were to just like inherit from the generic class and not change anything it would try and do terminal with xero it would try to go into configuration mode that's what h a init config commands does so you see we've got the init exec commands in exec in privilege mode it's going to execute these commands to set up the session before it sends in like your cli commands or your parser commands that you want so terminal length 0 that means we don't want it to have prompt for more we don't want paging we want it to just send the whole thing and that's that's what terminal length zero does uh there is no terminal with zero on dell and there is no line vty04 and there is no exact timeout command so uh i had to change these in the dell os settings to work there the state machine this is how it identifies it uses those patterns that we talked about earlier or no excuse me that's we're not there yet this is how it identifies when it should move from like raman to enable mode it should move from disable mode to enable mode this is how it determines uh what paths it should move through and the big one to look at that i did here is disable to enable that's user print user mode to privilege mode we identify that it's moving from a path of enable or of disable to the enable state with the enable word and then we're expecting a dialog to prompt with a password statement so the password statement says when we see the prompt we're going to give it some commands to issue and that's what comes up in the statements file i know to see it's already like we've gone down quite a rabbit hole you also notice this is inheriting uh even more configurations from the generic rp state machine so it's going to be trying to do a whole bunch of stuff when you look in the generic rp state machine look at this is generic rp state machine from here it inherits from state machine uh which comes from an even different place you end up going down a rabbit hole when you try and find out where all this stuff is long story short let's get back to where we are on statements so statements this is where i had to tell it when you get prompted for the enable password right down here when you see the pattern of password jumping back to patterns here's the pattern of password right there it's just password jump back to statements when you see the pattern of password the action is to run the enable password handler function now this is code that i copied and pasted uh from the generic section but here is the enable password handler where it sends in the enable password it calls the git enable credential password function which is where it actually parses through your test bed to find your enable password so that's the function that gets run here and that's how it gets logged in using the enable password totally honest with you here when you build your own unicon um you're probably going to end up like copying and pasting my code or copying and pasting from ios v you're going to run it it's not going to work you're going to do some troubleshooting the number one troubleshooting thing that i ran into is something that i should have already known the host name in the testbed file must exactly match the hostname of the device itself it must exactly match including case sensitive i ran into that issue i had to fix the hostname in my testbed file and then i started getting you know errors i got you know hung up on the password statement um and i was able to just control c to cancel it and look at the stack trace and see where it broke and then i just kind of went down the rabbit hole fixing one piece at a time until i got this to work so that is how the unicon structure works one last thing with unicon uh if you look over here in the folder structure down here underneath windows there is an init and you want to make sure you tack yours on your supported os uh to the init file there so at this point i've made all of my changes here to the code i go to github and i push my code into the actual repository itself back into github so here i am i'm back in my code um and this is the data knox unicom plugins from here i can go to pull request if i do no new pull request right here notice it automatically detects the base repository was from cisco test automation and it shows me what all has changed and what all i've added to this repo from here you click view poll request and it will actually tell you if there's any conflicts right here you can see it tells you it's able to merge and then you add comments in to your poll request telling them why cisco should pull your changes into the unicon repo so that sets up the unicon connection and at this point you're done working on unicon uh it's time to move on to the parsers themselves real quick if you want to find a quick way to get to the unicon plugins um of course it's in my code you can also google unicon pi pi and it will have a link there but yeah this is the quick way to get there it's right there if you just look in my code in my repository in the comments it's got this link right here and that'll get you there okay now to the parser this is the part you probably have been carrying about so under genie parsers uh i go to where's source src libs parser and now you gotta add your folder here so i added dell os6 so first thing is you need an init one more time uh init.pi and it needs to have this code in there from genie import abstract abstract declare token name so you need to have that make sure you've got your init.pi in your folder uh if you're already working with like an existing platform like ios xc it's already going to be there don't worry about it so from here now we're going to do we create our our code so there's really two components to building a parser first of all you have to have all of these from and import statements right here and there's two components the first one is the schema so i'm trying to create a parser for the command show version here so i create a class called show version schema it inherits from metaparser then i create a class called show version and it inherits from show version schema so the schema is what we're going to say the exact output should look like we identified the keys right here and then we identify what data type they should have in my case these are all strings if we look at other ones like show iprout i identify a key of routes then any which means this could be any the next key could be anything we're going to dynamically create that key on the fly but you can see i got string as a data type boolean as a data type and integers as a data type here so let's jump back to the show version so this sets up the actual schema of what i should expect here's how i actually did this i've saved the schema for last i created my parser first which is uh oops which is what we did down here and then i came back to it after the effect and created the schema so that's what i'd recommend you do is start actually with the parser so the first thing in the parser is we specify the cli command that is going to actually be run uh when the parser is called so that is that is the first thing you do is you set the cli command in the comments here i actually put what the expected output should be what the text should look like and this is what we're going to do the way this parser works let me draw here is it's going to issue this command show version and then it's going to loop over every line in the output and it checks regex expressions to see if there's a match on this particular line so if i actually scroll down just a hair i jump up just a little bit just going to jump ahead just a little bit right here you say four line in out split lines uh basically that means four line that's in this output it goes through and it says okay start with the p0 regex expression and see if there's a match on this line if there's not a match m remains null so if there is a match m will not remain null so we then go in to the match and we say like okay we need to grab the machine description variable from that result resulting set uh and put it in my result dictionary that we're going to be returning to python so this is kind of what that's kind of just the flow of it now let's actually dig into it so the first thing i'm trying to do is like okay i've got this machine description here this is the line and i want to grab this chunk right here dell networking switch that's what i want to return uh in my schema so right here i've got a schema called machine description and i want that machine description to show me the value of dell networking switch so i have to create a regex that matches this how do we do that because nobody's good at regex right we go to regex101.com and here we go so if i copy in this text output right here and i go back to this section let's just unindent this real quick okay now what you can do as you can see i'm going to copy this regex description everything inside the quotes don't copy the quotes if you're trying to work with regex101.com and i'm going to paste it in right here so you can see how this worked this parentheses question mark p creates a variable inside these little bracket brace line things here called machine description then since this is really the end of the line here i just said match anything as repeating times because i want everything from the end of the line from from after this space i want everything else and put that in this machine description so that way when this regex expression runs it's going to find you know starting with machine and then a space repeating as many times as we want then description then match on a period repeating as many times as we want that's all these periods then a space then i set up my variable to collect everything after the fact and that's going to be dell networking switch and over here you can see it's got the full match it matched this line and then it created the group match right here it matched the machine description variable uh to networking switch the same thing kind of goes for any of these lines here for instance let's grab uh the burned in address the mac address here so if i grab this line right here go back to regex101 paste this in you can see it jumps down here and it kind of follows the exact same pattern again where it matches burned and a space in in a space mac and a space address uh and then repeats as many times as we want um the periods and then a space and then everything after the space is the burned in address so this is how you set up your regex expressions uh and kind of use the regex cheat sheet i'm not good at regex i used a regex cheat sheet the big thing here is just keep in mind that this parentheses question mark p is what creates your variable that it's going to be looking for and then within the parentheses whatever comes after this little brace that's what it's trying to match on for that variable so all of these things we have uh for each one of these lines in this output where there's data that i care about i created a regex expression then it loops over each line and it tries to match that regex expression and if there's a match we use the group dict uh little parser function method thing here and specify the key that we're looking for so in this case p0 was matching machine description and when it matches it's going to be holding that machine description value in m group dict machine description now this line right here on 100 to 101 uh this is if version is not in version dict you see i created a version dict and a result dick uh what that does is this version dot set default version with an open dictionary that creates this first kia version with an open dictionary the result dict after that is everything contained within version so when you look at these parsers down here you see all of these things are setting result dick machine description result dict model result dict machine type that result dict is everything contained within version and i'm just setting the keys here so that's how i created the parser with starting off with the version key so the same drill here again as once you're done creating your parser there's there's a couple things you need to do you need to go into the change log and go into undistributed and you can see uh here's where we can go ahead and add just a change log about what we've made so if i take a look here it's pretty straightforward the format they all look like this see we just say what the vendor is that we're doing in my case it would be dell os6 we have a small description did we add something did we change something and then what is the actual change itself or the addition that we've done here so add your change logs in then once it's all done you commit it back to github back up here oh did i close my github github and you go back into your repositories and you go into genie parser and then you create another pull request so there we go uh here we go i'm again able to merge uh there's no conflicts there's my code changes that are all listed here um make sure you take your test bed out if you've accidentally got it in the same repo uh just take that out and send in your pull request for your parser so there you go that's how i went about creating the dell os6 unicon plugin as well as some genie parsers let me know what y'all think if i need to go into more detail on this um or if that was sufficient enough to get you started um yeah there you go

Channel: Data Knox

Views: 2,007

Keywords: pyats tutorial, pyats genie, pyats python, pyats framework cisco, pyats framework tutorial, pyats example, pyats documentation, pyats cisco, pyats parsers, pyats install, pyats unicorn, genie parsers, pyats unicon, pyats how to, pyats development, pyats docker, pyats genie parsers, pyats genie install, pyats genie documentation, pyats genie models, cisco pyats, network automation, cisco genie, network testing, cisco devnet, cisco python, python pyats

Id: 0WvizVReKLs

Length: 29min 54sec (1794 seconds)

Published: Sat Dec 19 2020