Creating a pyATS | Genie Parser from SCRATCH

recently i released a crash course video basically summarizing the work that i had done for pi ats creating the unicon plug-in for dell os6 devices as well as creating a couple parsers for dell os6 devices i thought it would be really cool though if we actually dug more into those parsers so what i'm going to do in this video is record myself working through creating a parser from scratch that we can kind of see what my thought process is and how i go line for line actually creating a parser so here's how we go here's how we get started creating a parser for genie in pi ats let's get going [Music] all right so here we go um we are going to get started off by working in my local containerized environment if you haven't watched i've actually got a couple videos now on how i've been doing the pi ats development and how you can get started doing pi ats development with the pi ats container so if you haven't checked out those videos i'm going to put links in the description so that you can figure out how i'm using my local development environment in general as well as how i'm actually doing pi ats development so here we go i'm going to fire up my local container where i've got my pi ats environment already built out as well as the git repos that i'm working with i of course forked the git repos so let me let's get this going here i'm going to connect into that container i'll say attach to a running container and there's pi ats dev now it's going to i've been doing some other work here so let's go ahead and kind of close these out and we'll take a look at what we got here so in my working folder um i've got my genie parsers and i've got uh let's see just genie parsers working genie parsers there's unicorn unicorns down here too and i've got my testbed device already set up here in my working folder so i can connect into them so if i look here my tester.pyscript this is what i'm using just to test things out so let's do this let's bring up the terminal and let's do source bin activate in order to launch the virtual environment with pi ats so at this point i'll change into my working folder and i'll just do a little sanity chest or sanity test i'll just do a dry run here python tester.pie and this should there we go so it looks like it connected in got the routes and then this is the parsed output right there at the bottom so i know that my environment locally is up and running so uh let's do this let's fire up the cli get connected to my dell os6 appliance enable mode there we go and here we go so let's start off with let's do something like i don't know show what should i do like show mac address table that could be a good one pretty easy what about show ip protocols also pretty easy i think i'm going to go with show mac address table that's what we're going to do in this one let's do show mac address table all right so under genie parsers i'm going to go into src for source then genie then libs then parser then dell os6 and this is where i want my parser to be i'm also going to have to create tests so let's start off by creating the file itself in dell os6 and it'll be show mac i don't know how they want to do this address hyphen table maybe dot p y uh let's check real quick and see if they've done this for like ios xe show nope not there let's see if it's somewhere else like catalyst 3850s nope not there what about ios no no show mac address table there huh i thought this would have been a more popular one to do well i guess let's just boldly go huh all right so what i'm going to do is i'm just going to copy the output i'm just going to copy one of these scripts just copy and paste and then we'll tweak them along the way copy paste there we go all right so let's start one line at a time schema for show mac address table quote there's the first thing we're doing next thing we've got the schema so this is what's going to define the output and i don't like setting the schema until i know what variables i'm working with first that's why i skipped creating the schema first so what i am going to do is i'm going to jump ahead to where we're actually building the parsers and grab these variables first and then we'll come back and build the schema based on the data this will make more sense in a second so i'm going to grab this output it's going to look like this there we go and i'm going to jump back here and down here under well first of all let's change the name of the class this is going to be show mac address table schema and then down here in the next class this is where we're actually building the parsers this will just be show mac address table uh here let's copy let's do this let's do show mac address table schema because that's what we pass into the class itself and now for the text parser for show matt address table on dell powerswitch os6 devices all right the cli command itself no abbreviations here this matters show mac address table and then as you can see right here this chunk is what the expected output should be from the cli command so i'll paste this in and i'll tab it just a little bit so it's nice and pretty and to go ahead and set myself up for success i'm also going to bring up regex101.com so i'll go to regex101.com and i'm going to put the test string in right here just like that now this is what we're going to start working with okay so the first set here is we're going to define cli we leave that alone we sleeve this if and else out we'll leave all this these first four lines here from if output is nine all the way down through the else out equals output leave that alone i'm going to set a main dictionary called mac table dict so that way up here this is what the mac table dict is going to be the result dict is going to be uh containing all of the results for each item here so it's going to look like this i'm going to create a primary key called max and then i have to decide how i want to handle this what's going to be a unique way to identify each mac address and contain the data in a clever way to me it you would think that the key would be the mac address and then it would contain a dictionary that specifies the vlan the type and the port that's how i think i'm going to build this so what we'll say is i'm gonna i'm gonna jump down past these regex express actually no i'm not we're gonna come back to that this will make more sense in a second yeah yeah i see where we're going now okay so all right so what we're going to do here is we're going to loop over each one of these lines and it has to match a regex expression now the question is do i care about total mac addresses in use or aging time do i want to return this data from the parser i don't think so i think i only care about the actual mac table itself right now so let's work on that all right so first thing i notice is the leading characters in the mac table are digits so if i say the first character is a digit repeating as many times as we want that matches my leading characters the next character oh and that's not what we want either we actually want to save each one of those as a variable so i'll open question mark and p and i'll call this vlan that's going to create the variable called vlan within the parentheses matching digit plus there we go so open parentheses question mark p specify embraces what i want my variable name to be as vlan then d matches oop matches digit plus says repeating as many times as we want because this could be 1 this could be 4094. so i have repeating digits as many times as we want then close parentheses and that's how it captures the variable then i'm going to say space as many times as we want and this leads up to the mac address so i'm going to open parentheses again question mark p let's do brace thingy again bia for burned in address and then here's how what we're going to do we're going to do w for either a character or a number i believe that's right plus repeat it as many times as we want an escaped period w again repeated as many times as we want an escaped period w again repeated as many times as we want close got it so now we've got the mac address then we have space repeated as many times as we want we're going to open another variable here because we want to capture the type p type and here i don't know anything outside of dynamic management or static is that one of the one i'd have to google it so here i'm going to open a brace and i'm going to put management type static let's see if that works it does not maybe it wasn't a brace maybe that was a parenthesis that was supposed to be parentheses okay so it captures management but not static regex cheat sheet range not in range let's see it's pipe on this list where's pipe no i've done this in the past when did i do this in the past let's look at the other one was it route ah yeah it was a parenthesis pipe and then a space okay oh i put static it's supposed to be dynamic oh so there we go just type the wrong thing let's delete these spaces out so outside of the parentheses we'll say spaces again repeating as many times as needed and my last variable here is going to be port and this can just match anything for the rest of the line there we go so now i have my regex expression i'll jump back to my code and i'm going to set p0 here we've set up the dictionaries that's going to contain the data now we're setting up the regex expression to match on so let me just delete this out and on this line here that says p0 compile there is my regex expression so oops wrong file [Laughter] here we go back to p0 for we messed that up that would have been terrible that would have been embarrassing okay there we go and now i don't need these other two registered expressions uh this is the only expression i care about matching on at this time so what it's going to do it's going to run the command it's going to grab this whole output here and then it's going to loop over it one line at a time and right here you can see this line p0 dot match so here's the compiled right expression we're using the match expression on the line that we're looping over so if the regex expression matches this line vlan here the one is going to be placed into a variable called vlan and the mac address is going to be placed into a variable called bia and the type is going to be placed into a variable called type and the port is going to be placed into a variable called poor so that's how we're going to do this so we could pretty much delete this chunk out here so on the first part is if so we're going to match the line if m is true then we've matched and what we need to do is we need to start setting variables in this case i'll put vlan equals m group dict because that's how regex returns these and we say vlan so now my vlan is going to be matched to my vlan is going this vlan variable is going to be equal to the group dictionary that's returned by the regex matching expression containing the key variable next line we have the mac address where the burned in address same thing here group dict and we're going to put bia next line here we've got type m dot group dict type report and there we go so now all of my stuff is contained within these variables what we have to do now is we need to set up the actual output itself so here's what i think it's going to look like the actual output itself first let's look at show ipro so i'm going to set up the main primary key that it's going to work with i'm going to copy these lines here that say if routes not in route dict set the default key to be routes with an open dictionary and that is held in a dictionary called result dick so let's copy this and put this here we'll now change this to be if max not in mac table dict the result dick is going to be mac table dict set to fall so what this is really going to look like at the end of the day uh is it's going to look like output just as a sample output it's going to start with max and then an open dictionary so what we've done here this result dict equals the result dict is a dictionary where the default key is max so basically what result dict is is everything contained within the key of max so now what i want is i want the burned in address to be the next key so it'd be something like this but it's not bia it's actually going to be the actual variable of the burned in address so here's what it's going to look like we're going to set result dict burn in address equal to an open dictionary so that's going to create the burned in address here with an open dictionary and now we can add the properties of vlan type and port to it so we'll say result dict bia we'll say vlan will be the name of this key is going to equal vlan and result dict bia type is going to equal type and result dict bia port will be the name of that key and it will be port deleting that out and cleaning that up okay so now we actually have the structure of how this is going to look so now we can jump back and create our schema so first off we've got max as the main key which we set then our next key is going to be the open dictionary of burned in address so the way we write that in the schema is kind of already stated here it's just any meaning we're able to accept any data or any string being the next key in our dictionary then we specify vlan type and port and looking at the output here honestly i think it's okay to leave all of these as string so i'm going to change this to vlan i'm going to change this to be type i'm going to change this to the import and i'm going to delete you and you and let's see i've got one two three open braces one two three close braces i think we did it okay so let's see i really don't know if this is going to work with that hyphen in show mac address table this is going to be it's going to take me for a loop let's change into genie parsers we need to first develop so i did make develop there now i need to make json and this is what's going to override the genie installation with the local genie folder that i've created here in cloned off okay everything looks good moment of truth show mac address table save it change back to that folder and run the script i'll be ta-da cool hard part's done now that's left is to create the tests so under the test folder here in dell os 6 i'm going to create a new folder called show mac address table and if you look at the other folder structures we've got one folder called empty and one folder called equal so i'll do a new folder called empty and right here i'll do new folder called equal and empty the files are always the same it's empty output output text so here i'll do new file empty output output text and equal if you look at equal we have the expected output which contains the keys guess what i'm just going to copy this right here ctrl c to copy that on google i'll just choose json i'll google jsonpretty there's a json formatter here copy this so i've got my pretty format and under my equal folder i'll choose a new file we'll call this golden output one expected dot pi i create a variable called inspected output paste in my expected output and there it is very cool the last file here to create under equal is golden output one output dot txt and this is our actual text output right here paste it in can unindent it all and save okay last thing we need to do is actually create the unit test itself so here we go you see under unit tests i've got test show test show to show that's what i'm going to do here so let's kind of contract these empty and equals and the folders here here and here okay under tests let's do new file test show mac address table and i'm just going to copy show if you route again paste it into show mac address table so here we've got the expected output and the golden output guess what we're going to copy this from we're going to copy all the way down here and my expected output okay so my golden output has now been saved oh we got to do some renaming don't we let's rename the class test show mac address table and let's change this text here test show mac address table and the golden output so i can close you now copy this paste this let's go ahead and indent it and last thing is change this definition mac address table it has the exact same name as the class that is correct uh so and then the object here show mac address table which means we need to import it would i name this again for us table oh i don't think it's going to like that i don't know what it's going to do with that hyphen in the name let's rename it and see if that makes a difference and change this to be underscore uh real quick we're just going to rebuild this make develop and make json i want to see if changing that underscore made a difference if it broke anything at all we'll rerun the tester python test er hi did not make a difference okay so underscore table works just fine i have my test now the last thing i need to do is i have to move this to the official test folder so what i'm going to do is i've got the test show mac address table highlighted here i need to rename this to have an underscore and i'm also going to grab the associated show mac address table folder i'm going to copy these and under genie actually yeah under genie parser there is a test folder and there's dell os6 and i'm going to paste these in nothing needed in it everything looks good and one final sanity check to make sure runs perfectly fine let's check in the code oh nope let's not check in the code do not check in the code change log we have to update the change log so under the change log folder we'll go into undistributed and since they haven't you know what i'm just gonna do a new file change log add dell os6 mac table today is 2020. i think today is the 20th 21st 21st well it is 306 rst just going to copy this one that has new set up right here in the center paste added parser show mac address table now i have everything set up uh so i am going to check all of these in just verify they all look good nothing here is altering an existing file that's what i want to make sure of looking good okay stage all changes no no don't revert all changes stage all changes uh commit message is add dell os6 show mac address table parser commit and push final check github.com slash data knocks repositories genie parser and i see right there add dell os 6 show mac address tape now here's the thing uh if you have not already created a pull request um like i have you'll click on pull requests and you'll choose new poll requests and this will uh show the comparison between cisco's genie parser and your current status it'll show what else changed between them um i have already created a pull request so my commits have already been pulled already a part of that existing pull request this one right here pull request number 365. so i don't actually have to create a new pull request right now it's already contemplated right there as part of this pull request that's already existing so if i actually go to the actual genie part this is cisco test automation genie parser pull request there's my add dell os6 parsers right there and we can see right there at dell os6 show mac address table parser has already been contemplated as part of this pull request so that's how i got this up and running and that's how it's now in the genie parser section so that's been how i work through creating a genie parser thanks for stopping by y'all see you in the next one

Channel: Data Knox

Views: 3,125

Rating: 5 out of 5

Keywords: pyats genie, genie parser, pyats cisco, cisco devnet parser, dell os6 automation, dell powerswitch automation, dell os6 parser, dell os6 json, dell os6 rest, pyats genie dell, pyats getting started, pyats framework cisco, pyats tutorial, cisco python, python genie, python pyats, cisco genie, cisco pyats, cisco pyats genie, cisco pyats getting started, cisco genie parsers, cisco genie learn, cisco pyats documentation, pyats framework tutorial

Id: knxkbWTamBY

Length: 30min 53sec (1853 seconds)

Published: Tue Dec 22 2020