Your First Python Project

ah there we go all right welcome everybody uh we're just gonna give a few minutes for those that didn't arrive early so just please stand by i'm gonna work on some things in the background trying to make the lighting just a little bit better for us there we go all right uh hello everybody uh we are just a minute past the hour and i honestly i don't know uh happy thursday happy thursday indeed uh i'm not sure how long this presentation will will go for today uh we've got a ton of different topics to cover uh and so this one will be just a little bit different than some of our previous sessions where in the the overall theme thus far has been that here we just we focus in on the basic constructs of network automation uh but we go really deep on those constructs and i'm taking that idea and dean and i'm just throwing it away because for today we're actually going to be covering a specific use case a specific situation that everyone will find themselves in very very quickly after getting started with network automation and that is you've got some kind of script right let's just say you have a python script that you've been executing and you you're ready for it to go live into production but you don't know where to go or where to where to look and so today we're going to be covering about uh half a dozen or so different technologies that will all play an important role in helping you get your production your production ready script into actual the production environment uh so we will again we're going to be touching on a lot of different things today however they will be a little bit high level each one of these things honestly requires a dedicated session within itself hopefully uh what you'll find is that this session is still going to be extremely beneficial to help you understanding what it is that you need to get up and off the ground running so we still have a lot of attendance coming in so i'm just going to wait stand by it for just a minute or so and then we'll go ahead and get started by the way while we wait for other friends to join in if anyone's got any questions or if there's anything specifically top of mind that you like feel free to unmute yourself at any point i'm used to being interrupted quite a bit so it's completely comfortable with it if you don't feel confident and sounding off if you want to just use the chat that's what it's there for as well hey calvin this is eric davis uh i have a quick question i know it's not directly related to the python part but um you know last session i kind of missed and i wanted to see if i could catch up um it had to do with the um ansible um session and um i've got i've got a working yamo file and i've got a um a script running an ansible but i keep running into um protocol uh what is it it says um error reading ssh protocol banner okay keep getting stuck on that okay uh what's the what's the end device that you're talking to um it's an ex-4300 the x-4300 okay uh and uh what uh how are you do you know off top of your head what ansible module you're using to call that ex switch yes um it is junos juniper.junos jennifer no sorry no that's the role sorry okay let me let me look at it real quick okay i had this working like a year ago and then i trashed it and i forgot all about it and then i tried to redo it again and i can't remember where the hell it was when the story of my life okay so the the module is juniper underscore junos underscore command okay okay so uh that module is gonna allow you to run like basically like a show command on a device and get the data correct i was trying to run show system uptime basically oh okay understood very very basic yep and do you have like a message of the day banner or something along those lines um on the device itself yeah like if you ssh no do it does it say you know welcome to no no okay uh so it definitely sounds like we have an issue either within the either within the version of the module that you're using or in the version of the code you should not have to worry about a banner um so um typically when i see that that type of message and the reason why i asked you what's the end device is that some of the python modules within ansible and some of these that are just ansible less they will rely on what we call screen scraping and that is basically a standard ssh connection from your uh from your script to the remote device and then the script will be kind of waiting right or i'm sorry the the the python code will be waiting to see some banner information come back from the device either like a message of the day banner or in sometimes it's just looking for the command prompt you know like user at switch and then um the less than sign or the greater than sign and so i'm i'm honestly i'm pretty surprised to hear you running into that on juniper devices because they especially using the official juniper modules uh like you are because that should not be a situation that you run into um let's do this if you're available um let's see i i'm trying to figure out if we can either uh have a dedicated session just you and i to kind of outwork through it or but i'm like also in the same mind realizing that i'm on pto next week and i'm trying my best not to do work in my team no it's okay i i actually am on pto starting tuesday through friday so okay well if if you wouldn't mind just shooting me an email and uh we'll we'll set up a working session between us and we'll get hashed out yeah that'd be great yeah i didn't i didn't want to like take up the time here but i just wanted to give you a heads up i appreciate it really do really appreciate it um all right so we got a full house um let's go ahead and transition into the overarching theme uh of today's presentation that is what should we consider when we're trying to take our automation into production and so i've broken this up into five different categorizations one is the the management of your script over its life cycle that's creating a project that's updating code that's you know deleting the project whenever the script has been retired or archiving it how do we do that what kind of services are available for us and then i want to touch on a second point that i think is incredibly important especially when you're working within a team construct with an automation and that's how can we work together on the same script without causing conflicts without overwriting each other's efforts or just creating some kind of clashing and so we're we'll talk a little bit about actually we'll spend probably a good amount of time about how these different services like github and git lab really facilitate a collaborative working environment for building automation then we'll touch upon some of the problems that people find themselves within this um this uh this problem is is commonly referred to as well it works on my machine type of issue right where the environment of your python script um on your local workstation will look very different across everyone else's workstation or any server that you're executing on so how can tools like docker and python virtual environments really help us have a consistent delivery for our automation also just really quick we'll touch upon some additional tools that will also help kind of make your life a little bit easier when you're working within an automation project and finally we'll touch a little bit on ci cd uh i i specifically led like a click bait thing does it really fit for network automation well uh we'll we'll touch upon that there's a lot of misconceptions around what cica cicd is and we're hoping to kind of clarify because there is a fit here for automation but it's a very specific fit that might not really align with the goals that you're looking for so this is the agenda that we have set out for us now again here we are we've got a python script we've tested it on our computer about 20 times we feel extremely confident that it's going to work exactly as we expect it to so the question remains are there any services available that will help us manage this script through its lifetime that's again the the creation of projects the collaborative efforts all those really great features that we need to make sure that we have a full history of the script over its life and so i'm going to be focusing this part of the conversation around two of the most prominent services that are available for us and those are github and get lab now in today's presentation i'll be using github but i just want you to to at least have this understanding before we go into this that the two are competing solutions they technically provide a very similar look and feel a very similar functionality for us and that is we create our automation projects as what we would call repositories and then kind of like in junos the way that we enter on a juniper device the way that we enter the configuration mode we enter this construct of this candidate configuration that allows us to stage any changes that we want to make onto the device and then there's a commit operation that goes through where we would see kind of like the before and after and we can give the thumbs up or thumbs down whether or not we want to push through with our commit well similar to that type of mentality in github and get lab we have this this construct of a branch and this is primarily the the place that we will be doing all of our work just like the candidate config think of it in in this way if i if i can draw some parallels here you're running configuration on a network device that is the production config that's what the device is running but when you want to make changes you you go into edit mode and then you start making all those changes to the candidate config that's exactly how we do things in or the best practice to do things in github and get lab where we have a a production version of the code nobody touches it just as it is what we do instead is we will create a branch of that production code which is more like a a let's just say it's a safe space right it's a sandbox for us to make proposals uh to make modifications and then before we actually merge it into the main production version of the brand or the main production version of the branch of the project then we'll have teammates collaborate on it we'll we'll ask others for review they'll be able to see the diff and then we'll actually merge it into the main branch we'll see a lot of this today uh and so this will again this allows us to work within this this kind of safe space without touching the production code uh it'll allow us to to you know refactor our automation or enhance it with new features now this pull request you'll hear poll requests pull requests i believe is entirely specific to github and other services that i've used like my personal preferences get lab again don't worry about them i highly recommend starting with github because it's fantastic um but in git lab and other services we'll call a merge request so if you hear poll requests and merge requests we're really saying the same thing and that is again that process that it gives us uh to review any kind of proposed changes that are made into an environment but this is going to be our our jump off point here for our automation actually i think i have no i don't i don't have another slide just yet let's go ahead and walk through the instantiation of a github profile and then we'll start to figure out how we can take our code our python script and move it into our production environment so with that being said i'm gonna open the wrong window let's start with this one here that one's good okay so i've got a brand new shiny email address it's called juniper automation oops i'm sorry i do not have an existing account i'm going to go ahead and create a new one so that we can see the entire process from front to start and i'm gonna turn off my dark mode just in case it's there we'll go and we'll create a new account now i i i wanted to stage this ahead of time but honestly the github registration is so cool i think everyone should be able to experience it just uh so let's go ahead and start with my email address of juniper automation at protonmail.com and we'll say yes and i'm going to create a very secret password here and they tell me that the password is strong my username is going to be well we'll just call it juniper automation there we are and do i want to receive any marketing information i do not so we'll just go ahead and type the letter in uh and okay so i need to start a puzzle so we'll just do this together uh pick the spiral galaxy that's gonna be this one and this one's gonna be this one okay all right so we should now have our github profile account oops i need to do a little verification so let's go ahead and log in now the thing to remember with a uh with uh github is this authentication piece now let's go ahead and just take care of this real quick four two nine three nine six four two nine three nine six and there we go uh we're gonna say it's just me i'm a student please thank you and i'm going to skip all of these goodies that we have right here all right and that's fine they're they're trying to customize the environment for me but uh we don't need any of that so here we are we've created a new github account now what's interesting to note is that github used to be exclusively a sas based solution and this was one of the differentiations between github and git lab but honestly today where we are is both github and git lab can be run on prem or they can be ran as a sas based solution in this case we're going to be using the traditional github sas solution that point though that on-prem off-prem point is going to it's going to significantly affect how relevant ci cd is for a slater just thinking of just we'll get to that point in just a second uh so now okay so we again we have a python script let me go ahead and pull that up so that we can take a peek what we're working with here i'm going to log into visual studio code this is the text center that i use for all of my automation development if you haven't seen me use this before i highly recommend checking this out this is probably one of microsoft's greatest products that they've ever made let's go ahead and uh well let's see if i can't reestablish my ssh connection i'm on the wrong network and if i drop off this is going to be a problem um let me see if i can't do something really quick i'm going to grab a script from a server i happen to be on the wrong dmz inside of my house so let's try to stand by just just take a second and that is a slash 16. now [Music] so i'm what i'm doing right now is i'm i'm using a back a back door and to get into my coding server development spawn do networking examples python and my first python program okay i'm going to clear the screen and then i'm going to chat out my app.pipe and this is going to be the python script that we're working with today now it might not look like anything just yet let's go ahead and make sure we got all of it and i'm going to copy and paste it into my text editor now this is all going to be available um it's already available on my my personal git lab so don't worry if you don't understand what you're seeing here i try to do my best to fully document what everything is doing but at the end of the day what this python script is enabling me to do is to uh to reach out to a pair of virtual firewalls and download their configuration actually i'm gonna i'm going to need to uh to be on the same local land let me let me try to not disrupt the zoom session but i i definitely can't deal with the being in the wrong dmz so please stand by okay i'm going to pretend like everyone can still hear me if you cannot please sound off i won't be able to hear figure out why let's go ahead and connect to host okay okay you can great thanks jason thanks eric i appreciate that small hiccup there i created a dedicated dmz dedicated vrf for all my smart home devices because i had some paranoia about apple and google and them listening in on me and unfortunately my computer joined that network and could not access the rest of my environment so let's go ahead and open up my automation example for python my first python project okay all right and i'm just going to open up my script here as soon as my page loads but again the idea behind this script is that it's going to copy and let's create a new page here uh this script is going to have the power to reach into any juniper device and download its configuration and store a local copy and that's what we're going to be doing today so as soon as i get some access here our very first thing to do is to create a project to hold this inside of github so i'm just going to paste it into my text editor for now and in the background we're going to create a new repository i'm going to call this repository my first python project now i obviously would appreciate if you had the capability of giving a little bit more descriptive name uh but for us our use cases this will be just fine now when we create this project or this repository in github we're gonna have a couple of different options for us before we go ahead and instantiate it one is to add a description and so we'll just say python script to download juniper device configurations right the next important really important decision do we want to make this publicly available for everybody or do you want to make this a private repository now back there was a time before microsoft acquired github where the private repository a free user could have one and honestly what i did is i just took one repository and stuffed all my projects in there and that's how i kind of got around not having to pay uh but honestly uh microsoft has done a phenomenal uh job of managing the github products since they acquired it a couple years back and one of the things they changed is the licensing and the user subscription so now everyone gets unlimited amount of public or private repositories now there's another couple of things that i always check when i build my projects up and that is we're going to add a readme file which is think of it as kind of like a landing page a landing document for people to read whenever they browse to your repository here you can add basic information about what your project is what are the dependencies of your project how does it execute examples documentation etc so adding a readme file right out of the gate is a really good start also there is this construct of a of a get ignore and what that is is it is a a plain text file that allows you to declare which folders and which files to not include in your um your backing up and to get hub so this will be really important when we talk about managing secret files i don't want my secret files to be stored on anywhere anybody else can see it so that would be an example of one of the files that i would include in a git ignore we'll definitely take a peek at that later but uh they're github also provides a lot of templates for these get ignores so i'm gonna go ahead and drop down the menu and i'm gonna say i'm coding this project in python and what the github will do is it will when it creates this git ignore file it will automatically include all the i don't want to say annoying but all the unimportant files that might come up in a python project so like python cache files and other types of things this git ignore file will automatically exclude them from the conversation and choose a license this is important if you're doing a public repository like we are i am not a licensing expert i'm not a legal expert i just typically go with apache 2 honestly because that's what i've been told to do so if don't take my advice on this if you're interested in learning about which license is appropriate from what i understand apache 2 and mit license are typically the most uh flexible and non-restrictive but honestly if you're if you're putting in a a project i would consult with the team can maybe even consult legal if you're going to make it public as to what kind of license is appropriate for your organization and i'm going to click this a green create repository now what this is going to do is it created those three files that we talked about uh the readme the license and the get ignore and we see that stored as the in the path of my username juniper automation and the name of my project or my repository is my first python project now i i want to pivot real just for a second really quick and talk about how before we we actually pull this down to our computer what this whole um get branches and and pull requests and what that actually looks like so let's go ahead and give an example i have these three files and one of these files is my readme file like i mentioned earlier this is my landing page for any kind of documentation or such when i open this up i have the opportunity of making changes but like i said just like with our candidate or with our running configuration we don't on juniper devices we don't just log in and start making changes to the to the production config we we actually will have a candidate config checked out for us where we'll do all our changes in a safe space and then we'll commit them into the production so rather than me making changes to this readme file right now what i'm going to do instead is i'm going to create a branch this will be a dedicated place for me to make any kind of modifications and then we'll review those changes before we actually commit them into their production so from here on the on the branch menu you can see the default is main and others the default is the name the word master don't worry about it just know going forward everyone's pretty much going to be using main as the primary production branch for code and what i'm going to do is i'm going to say let's go ahead and type in a branch here i'm going to call this branch readme update it doesn't have to really pertain to what you're doing i could have called this vanilla ice cream cone it would have been just fine but typically especially when you work in a team construct if you can give pretty descriptive branches pretty descriptive names for your branches then everyone else when they pull down those branches they'll understand what it is that they're looking at so i'm going to go ahead and click on this creep branch readme update and what i get from there is uh the top banner will tell me that we have now created a branch now i'm going to go ahead and start making the changes to our readme file here and this is just a standard text document it's written in markdown language md if you haven't ever seen it before highly recommend it it's just a much easier way of writing text um and so i'm going to go ahead and say i'm going to change the title of this readme to say junos configuration backup tool all right and this will say that that description is pretty good now anytime i want to make changes inside of and submit my changes into github i need to add a message and this is a really great opportunity for you to do your due diligence to document what it is that you're changing in the code so in this case i'm going to say we updated the title within our readme file okay and i'm going to this time i'm going to go ahead and select commit changes now this will actually commit my changes but only to the branch that's titled read me dash update if we switch back over to main we'll see that my readme file it's currently sitting as what we expect for our production which was my first python project but now i've got two separate instances of this file uh with and they're completely name spaced off from each other now let's say i wanted to take my changes that i just made to the readme and i want to add it back into the main line or the main branch of the code what i would need to do is i would fill out this process called a pull request and so within a pull request let's go ahead and move into this section right here called pull request and it github will tell me it says readme update has recent pushes and then how many minutes ago so what we'll do is we'll do a compare and pull request very similar to a show show compare on a juniper device and here we can see yes uh this message that we added we feel like it's a good message but what's important is that we can see a a diff between what was committed on or where it was before is highlighted here in red and what our con what our proposed changes are going to be are listed here in green i'm going to go ahead and select this create pull request because that looks good to me and then we will at that point have the option of performing a merge on this pull request when i do this we should be pro we should be given the option to delete that readme branch that we had created earlier let me see confirm merge here and it says pull request successfully merged and closed now they're giving me an option to delete that readme update branch uh we'll go ahead and make that so and we'll delete that branch now when i return back to my python project and i'm looking at the main the main branch the production branch i can see that that readme has been updated so that's this contract around uh github's pull request mechanism and a little insight into branching this is honestly a really really deep conversation to have around git uh probably warrants two hours within itself but i just wanted you to at least see that workflow before we get started because now what we're going to do is we're going to pull this for project repository down to our machine then we're going to add our python code to it and then we're going to push it back up and again i don't want to do this in the main branch so let's do this let's create a new branch here and i'm going to call this one add python script all right and we'll go ahead and do that now if it helps you represent or if it helps you visualize let me see i i know i've got a slider on here this is exactly what we're showcasing here we start with this main branch or sometimes it's also called master but then we create a dedicated isolated environment a clone of that as a new branch now this is where we perform all of our proposed changes to it and then we submit the pull request like we just saw this gives us an opportunity to bring in other teammates to review our proposed changes for them to sign off on it or for them to make recommendations we make those modifications as we see fit and when everyone gives the green light and thumbs up then what we'll do is we'll we'll go ahead and perform the pull request which will then take that additional change and unmerge it back into the production code so this is a slide that i thought i had it was just in a different section here when you open a pull request you're proposing changes to the code allowing someone to review it and again giving the thumbs up or thumbs down uh and then just like on the junos devices you can do a or the the pull request is going to show you a diff between what you had before and what you are proposing so with that being said let's return to our browser and now that we've got our own dedicated branch to do our to add our python file let's go ahead and figure out how to download this project to our computer the quickest way of doing things let's see we got a good question here how does the workflow happen in parallel our people are working on features how do we make sure that the changes are always in sync so a great question right we run into the same problem on on juniper devices like if there's two people ssh into the box and they're both inside of the config mode how does the how does the box know which one of the devices or which one of the configs to actually accept and in many cases and this is a default within automation as well the they they will enter a configuration exclusive mode on the network device and that's kind of how we handle these conflicts now in in contrast in this software world where we're writing automation we're writing code it's very likely that as i'm working on a file you're also working on a file and we might be working on that same file in different branches however as long as there's no conflict when i merge my code or i propose my merge back into the master branch as long as there's no conflict with that branch right there my code will be able to successfully merge and honestly if if we were all just working from the main branch and we weren't using branches you can you can absolutely do that it gets messy and it creates situations like the one that you're talking about uh but get the get protocol itself that we'll look at in just a second has the capability of identifying conflicts and will will actually prompt the user to say listen you're you're proposing this change to this file in this specific section and you have another teammate over here that's proposed this change and the in the same file at the same exact um at the same exact position within the code and so one of you is going to have to perform this resolution within git and there's a lot of ways of doing it um honestly if you're if you are interested in learning a little bit more please let me know and then we'll we'll do a dedicated session just on git and github and how to how to work them within this context um but i'm i'm trying to be cognizant that we do have a limited time but really good question because it's definitely a situation that you will run into okay so here we are we can see that we have two branches uh one is the main branch where we do not make changes to and then the other one is the add python script now the very quick way of downloading this is i could just click the green button and say download the zip file this um this will work but this is not the droids that you're looking for what you want to use instead is one of these three options that they have under this word clone now clone is a operation and get that that represents you cloning a project from a remote source it's a one-time operation for every project so we can see that we can we can perform the git clone over https we can use ssh keys and download it over ssh i know all of us network engineers just love that idea there's also a github cli tool for i think almost all the operating systems i'm pretty sure even microsoft's got their own they should considering that they own github but in our case i'm not going to mess with ssh keys although that is my preferred way of doing it because i hate typing in passwords for this case i'm going to go ahead and use a https so what i need to do is i just need to go ahead and click this little clipboard here it tells me that i've copied the the path so let's go back to our terminal and let me see clear the screen please oh i lost my connection whenever i changed my wifi so let me move back into my server go and i'm just going to create a new project actually let's go ahead and pull it down right here so to use to interface with github the interface will get lab we use a protocol it's an application protocol called get now this was written by the same man that created linux he's arguably which one's better i don't know but but linus the guy that created linux also created git and that's now kind of the de facto way of interacting with uh services like github and git lab so again i need to do a one-time operation of pulling down the repositories code to my local workstation so for that i do a git clone and then i paste in the name or i paste in the path for the url uh and then it went ahead and allowed me to do it and so we'll do ll and then i'm going to change into my directory my first python project we'll go ahead and clear that screen up here and if i look into this directory i can see that i've got two files right but it's actually three there's actually a hidden file in here so ls lsa i can see that i also have a get ignore file this was automatically created for us when we selected python as our language when we created this repository and there's our readme and if we open up the readme what we should expect to see is the juniper configuration backup tool which is exactly what we see all right so let me also type in git branch now this tells me that there's only one branch on my computer should be able to do a fetch operation let's see get branch uh it looks like we still only have the main so let's go ahead and create our own uh branch here that's going to be get a checkout branch and we're going to call this add python scripts we'll go ahead and do that and what it's done right now is it's gone ahead and it's switched me to a new branch on my computer called add python scripts so this command get checkout dash b uh get checkout branch and then the name of my branch will allow me to create that set safe space sandbox like you saw us do on the github website now for us i'm going to go ahead and copy that python script that i was showcasing earlier calvin r and we're going to say this is networking these are automation examples python and we're going to select uh my first python app app.pi and into my local directory all right so here we are we now have a python file app.pi we just take a peek at that again this looks like the python file that i was expecting to have let me go ahead and let's see the best thing for me to do now is uh is to help others with managing packages all right so before actually before we get to that let me go ahead and add this file and push it up into github so the way that i'm going to do that is i'm going to type in git add and then i'm going to point to the name of my file in this case it's app that will then tell the the get on my computer to look for this app file and add it into the the perspective of the project if you did not add it git will not be looking for that file so we added it to the to the to the world of get and now i'm going to do add a message now like i said before there is a requirement for you to add a message whenever you perform a commit operation commit is a way of saying hey we've made some changes i'm going to commit those changes right now and that's the second to last step before we actually do the push which will push it to the production branch but if i uh hit this right now git commit it will tell me hey look um you were supposed to add a message but you did not and so it'll put you in a text editor to force you to add a message i'm going to exit out of there and i'm just going to pass it as a as a flag here and say git commit dash m for my commit message i'm going to say add our python script and then close that and so now we've committed our changes to be pushed back up to the github repository we haven't performed to that push we've only committed those changes that's an important distinction to understand because if if we visited github right now we'll go ahead and do that if i refresh my page i will not see a an app.pi file we have not pushed it back up so let's go ahead and try a git push and then it'll tell us this is a one-time situation the problem here is that we created a branch locally on our computer but we haven't told github about how to we haven't told up anything about that branch so let me go ahead and copy the example that they have for us and i'm going to paste it we're basically telling github hey look we're working inside of a branch and we want to push this code that we just added into that branch up on github github says fine but who are you so in this case i'm going to say juniper automation and i hope i don't think that needs an email and my super secret password here and it says uh oh support for password authentication was revoked two weeks ago okay it looks like we're going to be using ssh keys after all so a really important thing that we learned today is that https is no longer allowed for you to use password authentication with github this uh this was not a thing for me because i still work inside of gitlab so in my case what i'm going to do is i'm going to add my public ssh key and i'm going to try to do this secretly where you guys can't see what's going on because if you had access to my get my ssh keys i'd be in a lot of trouble so in order for me to add my machine's ssh key what i'm going to do is on my profile in github i'm going to go down to the settings panel and i'm going to look for ssh keys right here there we go before i do that let me see if i can't turn off the dark profile i think that's pretty difficult for everyone to read i apologize for that appearance and let's go to day theme please please activate uh let's do single theme uh there we go okay hopefully that's easier for everyone to read i'm gonna go down into this ssh keys now i need to add my systems key so that it understands exactly who i am now on here this should be my this will be my private key now if you don't play in this space very often you don't work with ssh keys just understand that when you create an ssh key on your computer you're actually creating a pair a public key which can be shared with everybody because it doesn't really it's only a mechanism to help unlock but what you really need in the background is your private key so i'm going to go ahead and hit this enter key and look at my public key here and i think that's what it's looking for let me go ahead and paste it here and say add ssh key says the key is already in use so the key is in use on my personal github profile and that's going to be a problem for them because it doesn't show any kind of uniqueness so in this case i'm going to go ahead and create a new ssh key so ssh keygen we're going to say it's going to be rsa the bit is going to be 4096 and i'm going to add a comment and we're going to say automation monthly and i think i spelled that right let's go ahead and do that it's going to ask me where i want to put this key i'm going to say home my username ssh and then automation will be there i do not want to password protect my ssh key and well there we go now if i open up that ssh key let's say automation dot pub there we are and here we see our this is the public version of my key i'm going to go ahead and add that here and click add ssh key and so now it recognizes that key as associating to my username so anytime i want to authenticate with github for like a push operation or it's uh anything else it will now understand yes that is actually calvin okay uh so with that being said let me see if we can uh did i lose it i'm losing my computer here there we go let's go ahead and see if i can't authenticate with us to sht get at github.com and then i ssh automation see okay so what i just did is i did a test so t ssh-t github.com and then i passed in the path of my my ssh key all right unfortunately for my situation when i first cloned my project i did it over https and now i'm going to be trying to authenticate over through ssh so i need to make some modifications to my git let's do git slash config and i just need to change this transport right here to be uh let's see i forgot what it is let's come back over to my project we've got hitting backspace there we go all right and let me refresh the page get that light theme and so this is going to be i think that's right that should be right let me go ahead and let's go ahead and exit out i'm going to go ahead and check a different project just to make sure that we're not doing anything incorrect here networking automation automation examples python and we'll pick this one and we'll say dot get config okay it's the url is yeah it's get get loud okay so let me go ahead and edit my uh directory git hit config and we'll go ahead and pass this in here say url is this enter net okay now if i do a git pull let me see okay and now if i do a get push let's see if i can do this i've never used git push to the repository so let me final thing i'm going to do is i'm going to move into my ssh directory and oops ssh through the screen i'm going to move my primary public and private keys to a a new directory bar temp and let's see our attempt and then i'm going to rename my automation dot pub to id rsa.pub and so i i don't expect you to ever run into this situation i just have a conflict within my ssh keys and i don't really feel like spending right now working through that so let's do git push and get push there we go okay so what we've done in a very long method of doing it so we first tried to authenticate with https uh it seemed as though a week ago or two weeks ago github removed the ability to authenticate with username and password but so and that was a problem because when we were trying to push our changes up into github uh github was saying okay you want to make some changes well who are you and so i entered my username and password and said hey we don't allow that anymore so we defaulted back to ssh in my situation i had ssh keys already on my computer associated to a different account so i had to create new ssh keys and then rename my primary keys into the ones that we created i hope you never run into that situation but if you do you'll not have this video to to revisit about that all right so if i go back up to github now i get a message on my repository and says add python scripts had recent pushes less than a minute ago now you'll notice again here uh working from my main branch that i do not have in my app.pi which is my python script so but it is available in this add python scripts branch here now let me go ahead and see here like i told you earlier there will be a diff presented to you green and red in this case we see everything is green because we're adding a new file and so i'm going to go ahead and leave a comment and it says looks good to me and go ahead and create that pull request and this is an opportunity for ci cd to kick in whenever they detect that type of change and perform a bunch of auditing and actions on your script in my case i'm going to go ahead and just select the confirm merge request and this is or the pull request and it's going to take my app.pi and move it into my primary branch on my project so we come back over here to code we can see that we've got app.pi when we look into it we can see yep that looks like my python script looks good to us all right um now i've got a little problem here in that i created a couple of branches and i kind of got a little bloat so let me see if i can't go ahead and delete some of those branches i only want to be basing my work off of the main branch here so i'll go ahead and delete those two and come back over to here we should just have one branch that we're working for now on my computer my computer doesn't know that i just deleted those branches from github so i can do two things i can do a pull operation which will pull down any changes that have been made on the remote side to my local computer and once i can see okay there's been some changes into the main branch let me go ahead and check out now the main branch that's going to switch me over into the main and it can tell it tells me right off the bat it says hey look you're in the main branch but you haven't pulled down all the changes to main branch your two commits behind so we'll go ahead and do an ll just to validate no app.pi but now i want to pull down those changes from github so i do a get pull and that's going to pull down my app.pi and it tells me which files it pulled down and which changes to those files were made in this case we have all greens meaning that everything was net new inside of this script and if we run ll again we can see that we have app.pi okay so here i am now the problem is that i downloaded this repository has app.pi and it's supposed to be automating my junos configuration downloads but if i ran this right now python app.pi i'm going to get some invalid issue or i'm going to get some problems presented to me in this case it says you've got some invalid syntax well the problem that i've got right now is that i'm working within the default python environment and i can validate that just by typing the name or the word which and then whatever it is that i'm looking for in this case i'm saying which python and the system returns back to me which version of python or which path for that python file it will default to so in this case it's user bin python which i just happen to know is my system's default settings the problem here is that it doesn't have all those great python packages that i needed to be able to perform automation against my devices so i'm going to transition back to the powerpoint for a second to bring us into the next section here and that is how can we help maintain a consistent python environment if someone just downloaded my app.pi and ran it it's not going to work because they don't have all those dependencies that i was expecting to have on my computer so there's a couple of different paths that we can go down one is using this construct of a python virtual environment and that's going to be i'm going to be using a tool called poetry that helps me do this now what poetry will do is it will create a dedicated place for all of your projects python repository or all of its packages and dependencies and such and so the workflow in using poetry is you you type in git or i'm sorry you type in poetry and knit and that will walk you through an interactive way of saying which python packages that you would like this program this project to use and when you get through that little wizard the end result is it will present you with a couple of files that will say exactly which python project or which python dependencies are required for this project and then you can use those two files that were generated to then download and install them the value here is that one you get a dedicated isolated python environment just for your project but probably more important those two files that were produced from the poetry init those can now be included into my git repository and now anyone else that downloads my project will know exactly how to install my python proj uh my python dependencies and not just that it'll be the specific version that we're using so this gets away from the classic problem of hey it works on my machine i don't know why it doesn't work on yours this way we have exactly the exact python environment that's now being able to be replicated across windows mac and linux on any kind of environment so let's go ahead and walk through this process right here i'm gonna go back to my terminal and i'm gonna create a new poetry environment here so let's clear the screen i'm gonna say poetry init now i said this is gonna walk you through an interactive install for or an interactive wizard for you to create these these text files for you so the first one it's gonna default the package name to your project i'm okay with that it's gonna ask you which version this can be version controlled i'm going to just leave it at its default description i'm going to say um junos configuration backup tool and the author is going to default to whatever is in your git config files in this case it's my name my email address and my license here is apache 2.0 and compatible python version so you're not only depending you're not only pinning the specific python package versions you're also pinning a specific python version so in my case i'll say anything that's python 3.8 and later will work for us so we'll say yes now here's the part where they'll ask us to type in the names of our python projects or python packages that we want to use the very first one that i want to use is going to enable me to build api connections to my juniper devices so i'm going to say junos ez and c and that's going to be that represents the pi easy project now you can see we get a list of 10 different items that come back to us this is more like a search function and they're just saying hey we think you mean juno's easy and c but in case you didn't here's some others that we think are kind of in line with what you're looking for in my case i'm gonna say yes i was looking for junos easy and c so i'm gonna press the number zero and it'll ask us if we want a very specific version of this uh package i typically just hit enter just let it pick the latest now it's gonna ask us would you like to add some more i'll say yes i will i'm gonna add jxmlease this uh this is an easier way of using xml within juniper i'm going to go ahead and select 0 again and i'm going to go ahead and say keep the latest so we're good there and i'm just going to keep hitting enter until it's done and this is what they have generated for us this specific file right here this lists the python version and the two packages that i'm looking to install here so we'll say yes that looks good to me and all i did i'll type in ll here all it did is it created this project uh pi project.tamil file and a another hidden file called oh it didn't do the lock yet because we haven't actually kicked off the install so if we look into the pi project it's going to look exactly like it does up on our screen here just stating again what kind of dependencies we're expecting now if i want to go through with the download and then the installation of these packages into my new virtual environment i'm going to say poetry install that's going to look at that toml file that we just took a peek at and it's going to say okay in order for you to install pi easy you need to install these dependencies and as you can see these dependencies get automatically mapped out and and poetry will go ahead and download those for you uh now for me to enter into my new python virtual environment i'm going to say poetry shell now when i do that it's going to automatically create a dedicated virtual environment and it's going to put me inside of that virtual environment and i can validate that if i type in the command pip3s to look at which python packages i have installed in my current environment what i will see is things like junos ez and c j xml ease and all those other dependencies i didn't have to install these manually myself poetry already took care of that and again the power here is that it created that tamil file which everybody now can now download and automatically recreate my environment now i just happen to have a couple of additional packages that i need to have installed for us to be successful so one of those is going to be called python.end and we'll talk about this in a little bit but for now i'm going to add a new package into my virtual environment so for me to do that from the command line rather than going through that initial wizard and running through it again i'm already in the environment so we'll just say poetry add and say python dot and i believe that's the correct one and not only will pull down the appropriate python package it also updates the uh the tumble file so anything that i add later on will automatically be included into this uh this tamil file which will allow us to again make sure that we have perfect consistency there's one other thing that i want to add and so we'll just go ahead and say poetry add invoke and we'll talk about invoke in just a little bit but again i'm just going to go ahead and add it to my repository now if i try to execute my python script we'll say python dot pi it's going to say please enter your device's hostname or ip address now here would be a great point for me to just go ahead and type in either a hostname or an ip but i also inside of my python script i enabled a user to pass in a host from the command line so my case i'm going to say the host will be virtual srx1 and it tells me that i have some authentication issues and let's go ahead and pull out our our python to say sure enough this is the issue the username and password are supposed to come from my environment but i do not have an environmental file named username or password this is where that python package that we just installed called python dash dot inf really comes into play for us it allows us to create a an environment file that will host secret information like my username and password uh but uh and that will be imported into my script rather than me hard coding my username and password this enables the type of workflow to where everyone on the team has a an environment file on inside of their local workstation and it is automatically it gets excluded from from github entirely so we'll have to make sure that that parts working as well so let me go ahead and create that inventory file right now i'm going to say touch n dot env which is the specific file that the environment file is supposed to be and we'll go ahead and edit that right now and i'm going to just pass in here username equals and we'll say root and my password equals juniper123 all right should be good to go now let's go ahead and try to run this again and what we get back now instead of an authentication error is it tells us that we have successfully downloaded the configuration for this virtual firewall and it automatically put it in its own directory so if i clear the screen here let's just go ahead and do a tree let's take a peek at our new of structure of our python project we have the app.pi that's the thing that's actually running out grabbing the config and downloading it and then any kind of update or any configuration is automatically put into its own respective backups folder and this and it'll be sorted by hostname and then time stamped with the the day of the year and the time so that's the the functionality of my python script if we did that same script in this case for virtual srx2 it's going to download it and put it in its own new directory and you can see it's time stamped and such and if i wanted to get fancy i could just do a diff between the backups we'll compare virtual firewall 1 with the config that we downloaded for virtual firewall 2. and here we get a a typical linux diff this is what's changed in between the two we can see that the two have different host names different ip addresses they have an ipsec vpn tunnel up against each other so they're sharing different routes but anyway so we know that we've got those two configuration files let me open up this tree again all right so now this is the look of our directory now think about this how do i get these files and these directories now pushed back up into my my my production main branch let's first check out which which branch we're working on so git branch i can see that i'm on the main branch and i don't want to make those changes here so what we'll do is we'll we'll just go ahead and create a new branch i'll say get checkout branch and then we'll we'll say this is going to be add add dash backups and let's go ahead and do an ll again so although i just created a new branch again all my files are still there for now what we need to do is we need to add all those new files and check them in to get now i don't in order to do this we do the get add operation but i don't want to add every single file individually so i have a shortcut here where i can do git add dash capital a and this will add all of my files in my current directory into github but there's one file that i wanted to have excluded and that was my environmental file that had my username and password i don't want this to be stored inside of github so this goes back to that one of the very first files that you had created called the get ignore file and you can see when github created this file they created all kinds of things that are kind of the defaults i'm just going to go ahead to the very top and let's go ahead and insert another file to have excluded let's see so coming back up to the top of the file and i'm just going to type in env that's my environmental file that has my username and password i want to exclude git from ever looking at that now that we've updated the get ignore file to include the end we'll go ahead and run that kind of wild card of get add dash capital a for all files now we need to add a message you will get very comfortable with this workflow add any files that have changed add a create a commit operation with a message and we're going to say in this case store backup files all right and now we're gonna say get push and i'm expecting to get a message come back says you're you're looking to push to a branch that doesn't currently exist in github and which it does and so it prevents us it presents us with the message let me copy and paste it right here okay and what we can see is that we have uh five files have changed uh we've created uh the backups file the poetry and the lock and now we have successfully pushed that up into the github branch so any other teammate that's working on this will now get a banner message to say hey look there's been a new branch and there's been some changes inside of there so let's go ahead and click on the compare and pull request and let's look at these changes that are being made so one i can see the user on this in this case it was set to myself i can see any changes and so on the get ignore file the only change that we made is we told it to ignore our local info file so really good here in the backups we can see all the configuration that was downloaded from the device for virtual srx1 you also see it for a virtual srx2 and i think that's going to be oh we got the the poetry lock and the tamil file so i told you that when poetry runs it's going to create two different files one in this case is the tamil file which just lists the version of python that i'm using in any packages that i'm using there's also a poetry lock file let me just go ahead and show you it's gonna have all the specific metadata regarding all the packages that were we're running honestly you don't need to add the lock file um this will automatically be created whether or not you do it yourself the only one you would really only need to keep a consistent working environment is this project tamil file right here okay so uh these changes look good we're just gonna go ahead and sign off on them uh let's say looks it's mean and we'll go ahead and create that pull request again we're only making changes in this construct of a branch once we review those changes then we say yes or no as to actually moving them into the production branch i'll go ahead and say yes and so now we will successfully merge that and i can now successfully delete the the temporary branch that we were working from this this is the workflow that you will find yourself executing quite a bit when you work inside of a team construct personally i should be doing this but i don't because i i just work by myself so i don't really have anyone to ever review or audit my code so i typically just do everything in the main branch but please don't be like me please follow best practices create branches check them out review them with your teammates and such uh just my situation is a little bit more unique um okay and now what's really great about this is that what we've been showcasing is that we've been making changes to this project as we go along right we've been adding files we've been making changes to files and such but you know what's really incredible about git is that it has this construct of providing an audit log for you so here i can see that this project my first python project has a total of seven times we've committed some kind of change to uh to the project let's go ahead and open that up and what we can do is we can actually do kind of like a john madden play-by-play exactly you started here now you're here what were all the things that changed uh throughout that and you can also go back in time to that specific commit within the project's history and restore something if if somebody had submitted some code and it broke and you needed to quickly revert this is going to be your get out of jail free card the ability to automatically roll the project back to a specific commit is highly advantageous when you're working with complex automation for sure all right so going back to the the powerpoint here we talk about poetry we talk about the workflow the goal is to create a dedicated virtual environment for all of your python packages for this specific project you initialize the wizard you walk through the wizard you declare your dependencies or not uh you install any packages that you declared in that wizard through the poetry install anything else later on that you say oh man i forgot about you know adding in vogue well then you the operation would be just a poetry ad and then the name of the package it'll automatically update the tamil file so that everyone else will be on the same page as you and when you're ready to enter into the python virtual environment with all of your nice shiny packages use a poetry shell operation for that and if you want to leave your virtual environment you do a poetry exit so this is great but how do i get even better than this well the option would be to use docker now we touched a little bit last session on docker and but we we focused on like the context of how docker can be used to build out virtual lab environments virtual networking devices how to connect devices build bgp with them and all those sorts of things this time we're going to be using docker to create a perfect little isolated environment that everybody can run everywhere uh and have no fear of any types of dependency packages issues or or anything along those lines so not getting too deep into docker let me just show you really quick docker and well when i say docker what i really mean is containers docker is a company they're highly revered for being one of the the companies that helped kind of create an easy on-ramp into the container world so just know when i say doctor i really mean containers there is a big difference between containerization and virtualization this probably isn't the right place or session to have these conversations we touched a little on it last time but at the end of the day what you do with doctor is you create a an image you create a a a virtualization environment and you you kind of snapshot it as this image and then you add any different files and dependencies that you want into this docker image and then every time that you want to run your automation you just summon that image and then you tell it what you want it to go do so in this case with docker what we're going to have is we're going to have a docker file which is going to allow us to create the instructions needed for docker to build the image to our specification and the way that we actually build that container image after we have a docker file is we type in docker build and point to the path of that directory in this case it'll be in our local directory so it'll be just a dot now uh when you want to run your container image you would use a docker run command followed by a series of different flags and that will actually run your docker container image if your docker container was to stop you can restart it with a docker start command um or if you want to just shut down a running container you can do a docker stop in the name of the container or if you're just looking to check the status of your currently running docker containers you can do a dr ps docker is again it's a really deep conversation it's an entire career it honestly is but these containers are the foundation for modern application architecture when you hear things like kubernetes it's exactly what we're talking about except that's more of a management construct around docker containers so in our case we're going to take our python script and then we're going to include it inside of a docker image and that's how we're going to be executing our python environments that gets us away from ever having to worry about python dependencies or anything along those lines uh brian asks can you touch docker run versus docker compose yeah absolutely so docker itself again is a it's a command line tool that um allows you to build containers run containers stop containers remove containers right that's the construct behind docker your container itself is or it should be laser focused on one specific task when you start to hear other parts of organizations talk about micro services that's kind of what they're talking about doing is taking an application that's got all these different components and breaking it into individual containers and believe it or not managing containers is pretty difficult it gets unwieldy very very quickly especially when your application has like a container for the database a container for the web app the container for the load balancer container for this container for that it's very very difficult to manage all these things because you'll be firing off docker commands all day and all night so enter docker compose and we won't be touching upon this it's we should probably do a dedicated session but what docker compose will do is it allows you to write a yaml file that tells the intent of what you would like your containers or how you'd like your containers to run you can tell a docker compose hey i want this container to always restart if it crashes just go ahead and restart it so it's always up or you can say hey build a virtual network between this container and this container so that they they can talk to each other or or maybe pass environments into these different containers it's basically a a management tool to help you manage the lifestyle cycle of your containers a very very cool tool all right so for us let me clear the screen let's take a look at our project again here we have our python script that's app.pi we've got some backup files we've so we know that our the script's running the license just came with our github project poetry lock and pi project dot tomml those are my packages dependencies that we created through the poetry tool and then the readme file is just that clear text file there's also a couple of hidden things here one is the inv file where it's got our username and passwords uh the get ignore is a text file with a bunch of files and directories to never include inside of git that way our repository stays nice and clean and lightweight and then there's another directory uh dedicated just for all git operations we edited this earlier when we changed http to ssh we had to modify the git config so that it will use the ssh transport all right so there we go let's go ahead and i'm going to copy in a doctor file that i created this morning because i don't really feel like building these live is is the most enjoyable experience so we'll say my first there we go and say docker file and i'll copy it to my local directory now when i do my tree i now see that i have a file here called dockerfile docker is expecting a file called dockerfile so if you named it anything else like um like wu-tang clan for instance instead you would have to tell docker when you run your commands where to find the docker file by default it's going to look for something called dr file so if i wanted to build this docker and container i would say docker.build and then pass in the path in this case it's going to be my local working directory so i'll hit the dot here so what this is going to do is this is going to um actually you know what yeah this is going to go ahead and build out my docker environment while that is running let me see if i can't actually uh take a peek at that doctor file real quick while this is going c dot at code one and we'll try to do my first point and docker file okay so let's go ahead and take a peek at this docker file as it's actually being built in the background here so what we can see actually i'll use them because it gets me pretty colors all right uh so here's a list of instructions for my container image the very top i'm telling it what i would like my conductor container to be based upon now there's a resource called docker hub where companies will publish all of their container images for you i looked at this morning and i said i just searched for the official python repository and i wanted something that was running python 3.8 and something that was running on debbie and linux so bullseye was the selection for me um and that's how i that's how i derived right here but this just tells docker all right we're starting from somewhere we're starting with the base python package that's based on python.3 and ubuntu or debian then i add a little bit of metadata just different things about the container that that i would recognize and help me this does not impact the build this is just helping the future calvin understand what it was that i was trying to do here so i'm adding some labels now what's kind of interesting is you see that i'm passing in variables into my container based on passing this key value pair so we call in the end the environmental it's a way of creating a variable for your system and i'm saying i want my poetry version to be 1.1.7 and this python unbuffered thing this is a a weird python caveat when working in containers so you'll probably always have this set to one now i do some basic installation i do a package update for ubuntu and then i install a package called gcc it allows me to compile things uh and here's where we go ahead and we install our poetry here you can see i'm referencing the the variable that we had created up here so this command right here run pip poetry uh and it'll pull down the variable it'll say uh equals 1.1.7 so i know i get the exact version of poetry inside of my container and so anytime you see the word run prefaced inside of a docker file it's basically saying once the container is is is is is up during the build process then run this command inside the container so this is a standard linux or ubuntu or debian type of update and install if none of this makes sense to you don't worry you don't need to you don't need to be an expert in this case um but after we installed poetry then i changed the working directory to home slash python inside of my python container i copy over that the the two files that we created from our poetry which was uh project pi project dot toml and poetry.loc i also copy them into the directory that we just changed into inside of the container i set uh poetry to not use virtual environments because doctor is already isolated there's no benefit to using virtual environments inside of docker so i run this command first and then i do poetry install and i add a couple of flags i say no interaction meaning don't prompt me like you would typically with the wizard uh and no a ncsi so i forgot what ansi is supposed to do i copied it from stack overflow don't judge me uh and the last thing i do inside of my python container build process is i copy over our python script and also put it in the directory of home slash python so when this is uh when this wraps up what we should see let's come over to here we should see every single one of those things that we just talked about but as a individual step right so the very first step was built from the python image and then all the label stuff that we did so every single line that was in a python or a docker file actually translates to a separate step inside of a docker uh build that's really important in the docker world because if if the docker build process has already seen a step before and it creates a unique hash for every step if it's seen that step before it won't run it again so this is a way of of increasing the amount or decreasing the amount of build time that's needed so basically we just built this container image i wasn't keeping track let's just pretend it took a few minutes to actually build if i run this again and i run it again and i run it again nothing has changed inside of my project inside of my docker file so docker's already seen all these steps and it won't re-run it again so very very helpful to you know simplify your docker interactions now if i type in docker images i will see that i've got a lot uh oh heavens which one are we working from um uh let's grab out um python let's clear the screen um i thought i'd create oh here's the python okay my first python project here it is right here um oh that's all right it's five hours ago okay so just just stand by for a second we'll we'll get through this um let me go back and say if i now want to actually sorry let me run this build one more time i want to make sure oh there's the problem okay sorry uh completely oblivious to this i just built a container image but i didn't tell it what name to give that image really really important if you're trying to to summon that image in the in the future so the way that we do that we don't call it a name in the container world we call it a tag there's technically two aspects to a container image there's the name of the image and that's you can think of that as your project and then there's a tag and you can have different versions of tags just like you have different branches inside of your repository so in this case i'm going to pass in a tag and somebody it says oh rest in peace odb so we'll do one for this we'll call our tag our container image odb and i'll just pass in a a tag of latest okay so you'll know that although we just created a new docker image with a new name when we went through the build process it was just right down to the end again docker has already seen all these steps so it just had them uh there was nothing new so nothing changes so now if i say docker images grep for odb i can now see i have a doctor image called odb it has the tag of latest and it was created six minutes ago that was the last time this container image was built and it's coming in at a whopping one gig all right so now i wanna run odb right so let's say docker docker run odb and latest and i'm going to say python app.pi so what am i saying here i'm saying docker i want to run a docker image and then i pass in the name of my image that is odb and then the tag of latest and then once that image is up and running i want to pass it a command and that command is python app.pi so let's go ahead and execute this and what we can see is that i got a message of oh okay there's a couple of problems here one that it's expecting me to pass in a pty to have a live interaction with the container image uh the second one is that we didn't pass in a username or password uh long story short when you start working with docker and you start building a lot of these docker images you're going to run into a reality really quickly that you're going to be typing in a lot of docker commands so this is going to be what it's it's actually going to be needed to run i'm going to say docker run slash it for interactive terminal and then i'm going to say the name of my container odb latest python dot app.pi so there's all these different types of flags that you're going to need to know i t is really really important i t again stands for interactive terminal but i think it's actually interactive teletype i don't really remember but it's basically saying hey we're going to run this container and it's going to we want to be able to interface with what's actually happening in the background now for our use case we're just passing in a python script and we're having it execute but when you start looking at applications as a whole you'll start to realize really quick well there's a lot of ports that i have to expose i have to do some port address translation between you know port 8080 to port 80 or port ford ssh or do all these different things and all of a sudden your docker run command becomes huge it becomes a real burden for you but before we figure out a solution for that let me go ahead and just go ahead and validate this we'll say virtual srx1 that's actually going to fail because i do not have dns working in here let's just go ahead and pass the password 105.241 for its ip address and we got some authentication errors because the python script did not get the environment variable file that we passed into it did it let's validate that i'm going to run this container again but instead of running pythonapp.pi i'm going to ask for the bash terminal and so let's go ahead and clear the screen now i'm actually inside the running container right now and you can see when we use that workdir and that on our doctor file to change your working directory in our case we changed it to home python this is now the place that we land when we're inside of the container so i have to do ls lsa i can see that my environment file the dot env was not included inside of here so that's obviously going to be an issue for us let's go back to our doctor file and i'm also at the end of this file it's copy and paste let's also copy over our dot env into the same directory and now that we've made changes to our docker image now we have to go through the build process again so i'm going to say docker build dash tag odb latest and hit dot and the execution should just take a second longer and it did and so now we should probably be a little bit more successful when we run our script because our in file is in there and let's type in the ip address 105.241 and okay there we go we see that it backed up the configuration if i did a tree here oh that's the problem i did the did the backup inside of the container which is not what we wanted so at the end of the day there's a lot of challenges that you have to kind of work around working with docker because it is this like dedicated safe space that runs just for a limited time and then it goes away so instead of using that let's do this i'm going to bring in a new tool into the conversation by returning to powerpoint here let's see uh additional tools to simplify our life so the one that i wanted to talk about we've already talked about python dot end again the ability to put secrets and passwords and variables in a text file and then they'll automatically appear inside of your scripts but i want to do something else i want to use this python package that we installed called invoke so you can look at invoke as a way to create your own cli and don't believe me on that that's not exactly what i mean but basically you can write a python script and you can say whenever i type in this command i want you to do this right and that this can be really really long and complex this will definitely make your interaction with your development and your python scripts significantly easier especially when you're running it in containers because again the the um the commands can get really really difficult to remember so what i'll do is i'll go ahead and use a shortcut here calvin r go to networking automation examples on and my first python project and i want to copy over a file called tasks.pi i'm going to move it to my local directory now let's take a peek at this task stop pi now this is python so if you don't feel comfortable and you don't understand it it's okay just understand what we're doing is we're importing the invoke uh package into this python script and from here uh we can build basically our own cli commands that we want to run so uh similar to what we had shown before uh i can pass an environmental variable i can create variables in this case i'm creating a variable called docker image and one called docker tag and let's go ahead and update these if you would let's go ahead and hit enter i'm going to remove that and replace it with odb and the doctor tag is going to be called latest we'll do that so that's only relevant for this specific file it's not relevant anywhere else i'm also creating a another variable called pwd that's for path working directory and i'm setting it equal to whatever the working directory is that i'm within this will be helpful when we start working with containers because i can easily say hey i'm in this directory go ahead and import all my files in this directory into my docker container and so here let's look i've got three different cli commands that i've created one i've created a cli file or cli command called build and whenever someone types invoke build this is what's going to happen it's going to run docker build dash t for tag and it's going to pass in the environmentals that we are the variables that we have created at the beginning of the script the name of the doctor image and the tag and then the path to the local directory so instead of saying dr bill dash t odb latest dot all i now have to do is say invoke build now if i want to jump into that container rather than saying docker run dash i t dash pty name of image colon tag bash whatever now all i have to do to to jump into the linux container uh document container is just type in invoke shell right so i've got shell and there's another one down here called a script and this will actually run our python script within our container but it also does a couple other things it'll it'll remove the container it does all kinds of cool things all right so let's exit out of this and let's clear the screen we'll go through and we'll first try invoke build and of course it just rebuilt the image now again i just typed in two words and that was sufficient for the myriad of different options that i was going to have to do if i did this by hand let's also type invoke shell so just like before now i'm directly put it into my shelf and if i type in ls lsa i can see that we've also added in all of our files that were in our current working directory we copied them into the container itself now if i type in invoke script that's mapped into my the to actually summon the docker container uh mount my local working directory inside of the container and also um run my python script in this case i'm gonna say 182 168 105.242 and that will go ahead and back it up now what's really cool about this uh the previous time we ran the script within the container i mentioned that the backup for the container happened inside of the container and once that container was done with this job it went away it died uh and so so there went my backup of the config but what's interesting is that when i did the invoke uh for my um for my task here if we look at the very bottom this line that i have selected right here this one is is is worth all the money in the world that dash v says do a volume mount meaning mount my current directory inside of the container and that's where the pwd comes in remember at the top of the script we said pwd is going to be equal to my current working directory so i'm basically mounting my directory as a readable directory inside of my container so that when my container finishes its job and it writes the file to the directory whenever the container goes away i still have that directory i still have that file just a little bit of magic within working in containers now let's say we've done this work we feel confident let's go ahead and check in with what's the status of our project here now from git's perspective it says well since you lasted your push you we see that you've made changes to these specific files we've created a docker file you've created a backups directory with this specific path and you've also created a task.pipe now let's check to see which branch we're currently working on i think it's going to be main uh add backups it's not uh is add backups still a thing let's come over here i'm pretty sure i deleted this one if i refresh my branch yeah i just got a single uh single one so what we'll need to do is come back over to our script here and we'll just say git add and i'll say all so that'll include these three files that haven't been tracked before we'll say git commit and i'll pass in a message we'll say include doctor file and tasks dot pi to simplify our life and we'll go ahead and do that and get tells me it says okay we're working in this branch called add backups and you're looking to commit these three files we'll go ahead and say get push and it will say okay we've gone ahead and done that on github we've created a new branch called add backups and it's going to have these changes that you just did we come back over to github now and i refresh my page i should now have two branches so let's go ahead and look into that and there's add backups it was updated 31 seconds ago let's go ahead and try a new pull request on that and we see the message that we had passed in through the command line earlier and we can see again green and red what's been added what's been deleted in this case we added three new files and we looked good that was that's the backup for one of our firewalls and there is our doctor container or doctor file all right so this uh looks amazing let's go ahead and create that pull request and this would be the opportunity for me to include somebody else for a peer review and i'll just go ahead and pretend like i am that person i'll go ahead and create that pull request and confirm the merge now it's going to take those three files that we had changed and merge them back into the production branch and i'll go ahead and delete this development branch that we're working on so if we come back over to our project we should have a single branch which we do we see we now have the backups directory we see when we passed an ip address we changed it into underscore so that linux directories don't get all upset whenever you put periods inside of the path uh here i can look into the file yep that looks like my config that's good to go so we feel pretty confident that now anyone else in my team can visit this project pull it down and get off the ground and running either with poetry creating virtual environments or just by using the doctor file and we used a tasks.pi to help with some of those commands so that others can others don't need to know if i want to run your container it's got to be dr run dash i t dash dash rm dash v blah blah blah no just get rid of all that simplify it down to a single word uh and then call that with uh the the command of invoke really really helpful way of simplifying the way that you interact with things so invoke is a fantastic project um now i think we're at the yeah yeah we're at the the sketchy part of this presentation um so the question must be asked is there a way that we can automate the testing of our code of our python projects before we ship it into production now this is a weird one uh and it's a weird one because there's a lot of misconceptions on here so this is a topic that we call cicd so let's talk about what ci cd is let's talk about what a lot of people have in the networking space that i've run into have misconstrued it to be and then let's talk about where it can actually provide benefit for us so cicd is a a cicdc tool will enable people to help build software and ship it and deploy it into a production environment now this means a lot if you're building a an application project or an application piece of software that needs to run on windows you need to build in a exe file if you're running it on mac you need to build a dot pkg file or a dot a dmg file or maybe an iso or if you're packaging it for linux or ubuntu you've got to create a deb file or a yum file on on red hat linux where i'm getting at with this is that traditional software developers they have to com they they take their source code and then they have to run it through a compiler and then they actually get a uh they export the result into different packages for different operating systems and then they have to run those packages on all different types of software versions and operating systems and it gets to be a really really really difficult and extremely time consuming process to do so what ci cicd does and tools that uh play in this space what they do is they allow you to create this concept of a pipeline where you can have a state where you can create these different stages uh that will help perform a specific task within that build and deploy uh pipeline process so they typically are like uh there's a build a process where you know where we take in all the code and we run it through a compiler and then we get this specific package and then we'll run it through a series of tests right we'll test it on mac os big sur and then we'll test it on mac os catalina then we'll test it on mac os snow leopard or whatever the version and then we'll do the same for all the flavors of linux and we'll do the same for all the versions of windows and then once we go through and all those tests are passed we'll go into a deploy stage where we'll actually package it accordingly put it in the right part repository for people to download and then there there's typically a validation phase as well where we can run unit tests across that code so here's the problem right and on network automation for the vast majority is going to be python based almost exclusively and this is coming from someone who really loves to use javascript the reality is that python is king kong in this space and it will continue to be for a long while python does not need to be compiled and i'm saying that with an asterisk because there actually is a small compilation that you will never see you don't interact it just happens by itself but there's no files for us to assemble like voltron and then perform all these different types of tests additionally when we look at cicde and pipelines the idea is how we help automate the testing of this code we want to make sure this code is legit before it goes out there the problem that we have in the networking space is that most of the ci cd tools are sas based solutions so let me ask you a simple question how is a sas based solution going to build an ssh tunnel to 10.10.10.1 it's not going to have that route there's no way right so we can build all this automation and it can be just so rock solid but if the cicd tool lives in the public cloud then there's no way for that automation tool to actually run the code and perform a successful uh or unsuccessful operation so cicd there is applicability but it really comes down to what is it that your project is trying to do are you building a web application like in python django excuse me or python flask yeah ci cd makes a lot of sense because that will stand up a web server it'll be listening on port 8080 then you can do a curl command make sure that it get a response but for those of us that are building automation to talk to networking devices the routing thing right off the bat is almost a deal breaker for many but like i said at the beginning of this when we talked about git lab and github there are options to run github github and git lab inside of your own corporate it and honestly i'd be amazed if there weren't half a dozen versions of git lab running in your organization already a bunch of shadow it work going on believe me almost every company i've been in uh or that i've worked for has got like somebody's running gitlab somewhere nobody knows about it right and those types of environments if you're hosting gitlab or github inside of your organization well you fix the routing issue because then the ci cd tool would be able to execute your automation to to ssh in 10 to 10.10.10.1 and prove whether or not it was successful but the vast majority again of all cicd is typically sas based because it's typically meant for real software developers that are building real world applications and they don't have these kind of constraints that we do in the networking space so again there is use case for it but it's quite limited on the networking side that being said um and and oh sorry this is kind of a real good diagram that i stole from git lab that will kind of show you where ci cd kind of fits into this so we already are familiar with the branching concept right inside of our git repository we have a main production branch and then we when we want to make changes we'll create a new branch we'll perform all of our code changes inside of that branch and then ci cd tools will listen in for any types of new branches being created or any pull requests being requested and whenever those types of triggers are discovered the ci cd pipeline will then pull up pull your code into an isolated environment do whatever task that you tell it to give you thumbs up or thumbs down on whether or not that task was successful and then it will update the uh the project back into your mole request and say hey this looks pretty good uh this looks like it passed our auditing so it should be ready for merging into production that way anyone that's reviewing your proposed changes has a high level of confidence that yeah yeah the the ci cd works so the the actual automation should also work although that again like i said that's kind of a big uh step once they review and approve it like we saw we merge back into the production branch and then there's this construct of continuous development which again repeats the process over and over ci cd can mean a couple of different things continuous integration like what we're talking about here continuous deployment which really helps with the deployment of real software i'm saying our automation is not really real software but don't take offense to that um it's it's just a completely different construct here but the deployment for people that are building um dot package files or dot yum files or those are really important that's where the deployment comes in there's also a continual development which will sometimes be replaced with continual deployment development deployment people call ci cd things different things all the time now before i give you some examples and we'll showcase some ci cd here let me just point out something that i think is a real common misconception people see these types of automated pipelines and they think you know i really want that for my network automation i want to have a series of steps that we execute in a serial fashion and i that's how i want my automation executed right that is not what we would call ci cd that's what we would call workflow orchestration and that's one thing that ansible does really really well right you you write your first task at top you write your second task at the bottom and so on and so forth and ansible will execute it in a a series of workflow events now stepping outside of the ansible construct workflow orchestration is incredibly powerful for network automation there's a lot of things that we would like to do when we're building our automation maybe we want to phone home to our network source of truth and get a list of devices and once we have those list devices then we want to then we want to log in and change all the s mp credentials and then when we're done with that then we want to send a message to slack to let everyone know that we were successful that's a workflow right that's certain sequences and certain events happening in a specific string or a specific order of operations uh but they're doing different aspects of it ci cd is kind of like that and the way that they've divvied up their their their different stages but try your best to not confuse the two because it's it's very easy to think cicd's doing workflow i want workflow so i need ci cd you probably don't you probably don't at the end of the day but let's go ahead and give you some examples of this so back up in here we have a dedicated uh actions section inside of github actions is uh the ci cd mechanism inside of github it's actually relatively new on the block uh believe it or not github actually did not have a ci cd before it would rely on you the customer to either bring something like jenkins to the table or travis or bamboo or circle ci or any of the other hundreds of different ci cd tools out there in contrast git lab has ci cd built in from the from the get go it's one of the reasons why i use git lab because holy moly it's super simple to use it's really excellent but in our case we're just going to stick with github since that's what we have github is giving me some some recommendations on some things that we can do here i'm not going to use these i had something pre-canned already so let's go ahead and pull that into our directory here so what i need to do first for github to find out where my ci cd pipeline is i need to create a file in a very specific directory so i'm going to say make directory dot github the dot meaning that it will be by default a hidden directory and then another one called workflows and now that that's here again if we do a tree uh i i won't see it because it's hidden but if i did an ls lsa i'll see now i have a directory called github and inside of there will be my workflows directory dot github okay and there's the workflows directory all right so now i'm going to copy over a a a test file that i had for ci cd uh let's go ahead and move into my directory with all my examples networking automation examples python my first dot github workflows name of my workflow and i'm going to move it into the directory called github and workflows boom okay so pretty uh pretty gnarly command i just had to run but now if i was to take a peek let's go ahead and edit that workflows file and see what it looks like here uh so github actions demo right so here again very very simple yaml uh formatting for our job we give it a name uh we say when we want this to run so on on a push on a commit on both on a pole how do you want to trigger this cicd pipeline once you've created that trigger then you tell it what you want it to do in this case i want this these sequence of events to take place on a ubuntu image and go ahead and use the latest ubuntu image that's what we're saying right here and here's all the steps that we want to have executed in this case i'm just printing out to the screen a bunch of different emojis and a bunch of different texts and we can also see that some environment variables from github are also being passed into here and so this really isn't going to do a whole lot other than go through a pipeline and print a bunch of output to the screen but if you were doing a like real software development this is where you do your compilations this is where you would introduce your unit tests this is where you would do all of the fancy things to make sure that your code is packaged correctly it's deployed correctly it's validated and it shipped out just like you expected all right so what do we do we've added a new file how do we get that back into git well the first thing for me to do is say git branch to figure out where i'm at i'm still on that add backups and i know for a fact that github we deleted that branch so let's go ahead and create a new branch i'm going to call get checkout dash b for branch and i call this one uh we'll call it ci cd yep and so we've created a new local branch and we've checked it out right and so now if i say git status um i can see that i've got one directory that we have not included inside of our project so let's go ahead and run git that add dash capital a that will include all the things git commit and add our message as into introduce cicd into our project and then we do a get push it'll alert me it says hey you're trying to you're trying to push your changes to a branch that doesn't exist on github here's the command that you need to run i'll go ahead and do that and it's gone ahead and pushed it up into the github branch now this is going to be what's pretty interesting for me because i've never seen this happen but i've never seen this because i don't use github very often let's go ahead and look into this compare and pull request and now let's say okay that looks good so we'll say looking real sharp there all right go ahead and create that pull request now what should happen is that we should have a oh i expected a uh the ci cd to actually instantiate let's see if it did it did it did it just it happened way faster than we were expecting so we haven't merged our changes yet into the production branch however whenever we did a pull request that kicked off the ci cd pipe pipeline and if we look into the checks part of our pull request we can see exactly what took place so i got all these different steps here within my my project just like we declared inside of our file here we we set up the job there's a bunch of environment variables that github will set into it uh here we just print out some echoes check out the repository but all these are happening in a sequential order if you had a hosted version of ci cd in your environment like a github enterprise or git lab you could very well have included a step in here to ssh into or run your python script inside of that docker container that we had built and then save the file and that would be your test if it everything came out good then github will give you a thumbs up now before we we go ahead and commit this i just want to show this github will tell you it says all checks have passed meaning inside of your cicd pipeline we have successfully tested all of those things that you're looking for and we did it in four seconds we'll go ahead and click merge pull request and that should go ahead and now incorporate the cicd pipeline uh into our main branch so now anytime somebody pulls down this project and starts to to go to town on it we now know as soon as they uh submit their pull request this workflow that we just built out this pipeline will automatically go through and perform whatever tasks were declared again in the automation network automation space this if you're using sass it's going to be limited applicability maybe you just have it build your container image just to make sure your container image works right that's a really good use case for uh for one round of the brigade but uh hopefully that has been uh beneficial for you let me see i think we're at the time we're just a couple of minutes over so with that um i i just want to revisit kind of what we what we incorporated today because holy it was a lot um what we showcased today is one how to safely store your projects in a remote uh environment in this case we're using services like gitlab and github to to hold the the actual source code as we're working on the project we talked about branching a lot it's a really big concept it requires honestly a dedicated session just for itself but we showcase how you can use the construct of branching to work inside of a collaborative environment with other teammates and not conflict with each other we talked about the reality when you work in python you're working with a lot of external packages and dependencies and anyone else that pulls down your project right out of the gate is not going to have an environment that looks like yours so we provided two different mechanisms to draw address that one we use poetry to create python virtual environments and then that creates the file that we can check into github or you can build docker images which would make a really compact docker image that anyone can pull down at any given time and execute successfully with a hundred percent your environment within that container image we talked about invoke uh the ability to basically create cli commands uh by uh creating a python file called tasks.pi and then just dumping whatever cli commands you want inside of that file we also talked about python.m which allows you to store your secrets or whatever variables you want in a local environment file and then pull those variables into your python scripts whenever you run and we talked a little bit about ci cd and what its actual function in the world is and whether or not it has applicability into networking space all right uh so we're five minutes over but let's go ahead and open this up and see if anyone's got any questions for us um let's see give calvin his flowers brian what's that about man uh don't know what you don't know what you're indicating there man um but yeah i i will accept sunflowers that's that's wonderful um all right i don't know if anyone um so jason asked any thoughts on get ea i don't know what that is um i don't uh let's let's do a google i'm going to pretend like you're not mistyping it but let's see what this is hey calvin it's jason oh jason hey yeah hey so get t is like a open source ford software package it's a lightweight version of like github and gitlab so i was just thoughts on it because i know a couple of my friends who host this on their you know personal networks because it's way more lightweight than git lab and github yeah i i highly encourage you to check it out um i'm saying that without me doing any due diligence by the way but uh i uh listen i've run git lab inside of my home environment for a couple of years and um as much as i like the project it's really heavy it's really really heavy and i'm imagining that github is in the same position um they are both extensively heavy applications and when they go bad things can get out of control really quickly so i'm i'm all about testing out other alternatives and this one if it's as you uh have suggested here uh i mean just looking at the sponsors if these people are actually using it then it kind of speaks to um the credit for it because i know all these people and they're all amazing on our um because we have a or in my environment we're on a closed network we have git lab but we also host uh get t iron within the environment for various reasons but more so because it's super lightweight and uh especially with uh authentication stuff we don't have to really worry about it ah fascinating yeah yeah that's really interesting thanks for sharing this um yeah i'm i'm really interested in it um i would say roughly 99 of the projects that i build and work on i don't they're not public i don't share them with anyone uh not because i don't like you guys just because it's just stuff that i'm i'm kicking the tires with and not really needing um uh public access so this is is is actually really really intriguing to me um i'm imagining that you would be able to incorporate like a ci cd tool if you went down that path like i'm thinking like if i'm hosting it then it has access to my internal network so there might be some cicd that i want to do but i'm imagining if this is as lightweight as it says then it probably doesn't have that type of functionality which is honestly it's fine by me yes so not to go into all the details and and whatnot but within our environment we have a git lab and then we set up our own get t and then we're cloning repos from git lab that we want and then we're just hitting get t and as opposed to git lab and we don't worry about authentication yeah makes sense it totally makes sense i love to hear stories like this because it's uh it shows like the uh ingenuity of people kind of like working around uh you know complex systems so really cool story thanks for sharing yeah i i'm kind of interested in giving this a shot ah let's see um uh oh brian it means give you praise for the grace yeah thanks brian uh my first session was wonderful oh thanks abby i was really kind i really appreciate that um let's see uh the jump but thanks for the gracie yeah uh so again everyone will we'll go ahead and conclude this one um i will upload the uh the session up to to youtube um [Music] all of our past sessions are up there uh if you have any suggestions or anything that you're interested in learning please feel free to send me an email or shout out my way i'm open to suggestions if you want to learn more about doctor if you want to learn more about git or if you want to just get back to writing python or ansible please just let me know i'm open to any suggestions with that being said thanks again everybody i really appreciate it uh your participation today kate's oh no oh jason wants kubernetes um kubernetes is one of my favorite products and oh my goodness is it complicated it's amazing but it's so difficult um yeah we'll we'll see if we can't do an introduction course or at least a comparison between kubernetes and some of the other products out there like a personal favorite of mine is hashicorus nomad uh or docker compose that's another abortion uh kind of but yeah yeah all right uh well thanks everybody we'll go ahead and put this up to the youtube and we'll see you guys next month thanks so much have a good one

Channel: Calvin Remsburg

Views: 354

Keywords:

Id: GHujl7c_-hg

Length: 134min 47sec (8087 seconds)

Published: Fri Aug 27 2021