Using Ansible "Pull" Mode to Dynamically Automate Server/Workstation Builds

hello again everyone and welcome back to my channel in today's video i have a special treat another video about ansible which is one of my favorite technologies i love it so much that i've recently done an entire series about it that you should definitely check out if you haven't already done so now you don't have to go through that ansible series to understand this video this one is standalone if you have gone through that series well then i guess that'll be even more helpful because you'll get a deeper look at ansible in that series but this video again is standalone we're going to be talking about ansible pull and i've been looking forward to showing you guys ansible pull for a really long time i actually like it quite a bit so that's what we're going to go over in this video before we get started i want to send a special thank you to all of my patrons out there who have already seen this video because my tutorial videos are released on patreon first before the general public so i definitely appreciate each and every single one of you so without any further ado let's go ahead and get started [Music] [Applause] [Music] so what is ansible pull and why would you want to consider using this now first of all ansible pull comes with ansible so if you have installed ansible on a machine then you already have the ansible poll binary there's nothing new that you have to install and what it does is it allows servers or even workstations essentially linux machines to pull ansible playbooks from a git server and then run them locally even though it's downloading from a git server there's typically no need to maintain an ansible server and there's generally no single point of failure now it is true that if you run your own git server that does become a single point of failure but generally speaking you'll probably use something like github or git lab and that's maintained by you know another party there's a very little chance that those services are going to go down anytime soon now best of all it leverages the functionality of git to extend ansible now you don't have to know git very well to use ansible pull basically what i'm about to teach you in this video is everything you need to know but if you learn the more advanced features of git such as branching for example you can easily make that a part of your ansible configuration as well but let's go ahead and see what that looks like from a diagram perspective that i think will really help you understand the value of ansible pull so here we have a standard ansible layout we have an ansible server and the ansible server could be a workstation it's essentially a central point where ansible jobs are being run from in this case ansible server is using ssh to connect to each of the machines that it manages and as you see here it's managing your laptop hypothetically as well as two servers and that's all well and good but what if you actually have some instances that are not available now here let's just say for example you put your laptop in your bag because you're done for the day so you close the lid which makes the laptop suspend it's no longer on the network it's in your bag and also we have a server here that is not accessible now if you are thinking like an administrator you're probably thinking well we need to find out why that server is down but in this hypothetical example let's assume that we want that server to be down maybe that's a server that is only accessible during certain hours that helps with security and it also helps with cost savings as well because why run something 24 7 if it's not being accessed 24 7. so let's just assume that server does go down at the end of the day when everyone at the company is done working the server in the middle is going to stay on all the time now in this example ansible is going to reach server 1 and configure it everything is fine but when it goes to reach the laptop well it's not going to be able to do that because that's in your bag it's going to send you an error and then it's going to go ahead and try to connect to server number two it's not able to do that because it's the end of the day and that server is no longer on the network it's going to send you an error about that as well now there's multiple ways to work around this problem but what ansible pull allows us to do is think of things in a more dynamic perspective now here we have ansible pull you'll notice that the diagram hasn't changed all that much now the ansible server at the top was replaced by a git server that could be a get server that you've created yourself it could be github gitlab or whatever you use for your version control now what you will notice is that the laptop as well as the two servers are working in the reverse direction they are checking in to the git server they are running ansible pool locally so what's going to happen is the laptop will check in to the git server it'll download the get repository that has the ansible playbooks inside it and it will run it against itself the git server is not connecting to the laptop the laptop is just pulling the code to itself to run hence pull server 1 and server 2 well they're doing the exact same thing so what this means is that if you put your laptop in your bag you're done using it for the day you're not going to get any errors because the laptop itself is running the ansible job and as soon as you open the lid and continue working then the laptop will continue to check into the get server while it is online the same with server number two and that is you know one of the values of this scenario i mean you could even have something like an rv where you're going down the road and you don't even know if you're going to have internet access half the time and then your machines can simply check in anytime they are scheduled to run and if it doesn't work out well then that machine will check in again later on and try again and run ansible against itself it allows your environment to be more dynamic and then as an administrator when you implement a new change to the ansible repository then when the machines check in the next time they will download that change and then run the change so it basically allows you to have all of your machines basically run ansible jobs anytime you make a commit to the repository and that is really awesome i love ansible pull it's actually my favorite method of running ansible so now let's go ahead and set up ansible pull in the next section of this video i'm going to show you how to create a git repository and then after that we'll get it installed and i'll show you a few examples [Applause] [Music] for ansible pull to function it needs a get repository in this example i'm going to show you how to create a repository on github now if you use something else like gitlab that's totally fine but in this example github is what i'm using so that's what you will see in this video so first of all i will create a new repository and i'm going to call mine ansible pull tutorial simple enough i'm going to add a readme file you don't have to but you know just so we have something in the repository i guess notice i have it set to public and that's what i'm going to leave it as now you can use ansible pool with a private repository but it's beyond the scope of this video just know that it is possible to do that so i'm going to leave it as public and then i'll click create repository alright so now we have the repository right here it doesn't really have anything in it quite yet but it does exist so at least that's something now what i recommend you do is add an ssh key to your account to make this process a lot easier now i've already done that on my end i already have a key here but in case you don't i'll show you what you need to do so what you can do is click on your icon on the upper right hand corner go to settings you can go down here to ssh and gpg keys in my case i already have a key right here but if you don't already have that you can click new ssh key and give it a title here so i'll call it j ansible test that's good enough and then you can paste your ssh key right here if you don't already have an ssh key then in a terminal you can run this command right here ssh keygen so i'll just press enter and then to keep everything simple i'm just going to accept the defaults for each teaching you guys all about ssh keys is beyond the scope of this video but we only need a key so i'll just keep pressing enter we should have a key so if i go into my home directory ssh we should have at least these two files at this point id underscore rsa and the same but with the pub extension now this file right here is the private key we don't want to expose that at all but this public key right here that's the one we actually want to give github because that one is well it's a public key it doesn't matter if other people see it so we just take a look at the output here so i'll just go ahead and cut it out and we have this long string of text right here it's an rsa key this is the key and then this is the title this is actually my username and then my machine name is unicorn hence the unicorn in my batch prompt so what we're going to do is copy all of this to our clipboard and then here on github we can paste it in just like that so i'll click add ssh key and now we have that key right here so now what we can do is pull this repository on the machine that we are going to be working from [Music] so i'll just go back to the beginning i'll go into our new repository right here i'll click this green button make sure it says clone with ssh and then i'll click the clipboard icon to copy to the clipboard here but first of all we do need to have git installed on our local machine so if you get user bin git when you execute which get then you already have it if you don't then you're going to need to install it if you are on debian or ubuntu it's something like sudo apt install git just like that that will go ahead and get that installed but if you are on a different distribution you would change apt to dnf yum or something like that so anyway so now that we've added our ssh key we should be good to go ahead and pull down the repository now i'm currently in the ssh directory so i don't want to pull down the git repository here so i'll just go back to the home directory and then i will do git and then clone and then i'll go ahead and paste in the url and there you go press enter and there we go we should have the repository locally so i should be able to change directory to be inside that so we have the readme file right there so i am going to go ahead and edit that i'll use nano i prefer vim but nano is easier to explain in a video let's edit that just to show you guys how to make a change basically so i'll go down a line here and i'll just put a simple description or whatever you want to do and i'll just do ctrl o and then enter to save it ctrl x to exit out then we can do git status we see that we have a pending change right here so to add that change to the commit we could do git add and then the name of the file and then git commit with dash m and then a message easy enough and then we can go ahead and send that commit up to github with git push origin master so now if we go back to the browser and check on the repository i'll refresh it you can see that the new text that i've added right here is present in the file so we have successfully added our first commit to our repository and now we are all set to get started on ansible pool we'll get back to the video shortly but first i want to mention my sponsor lenode founded in 2003 lenode is the largest independent cloud service provider built on open source and lynode is giving learn linux tv subscribers 100 in credit when you sign up today if it runs on linux it runs on linux multiple distros are available including ubuntu centos alpine and of course arch and multiple server configurations make any app or service flexible and scalable use your linux server to host a website set up your own personal vpn create a next cloud instance host a game server and more linoad also features 24x7 by 365 support that's available by phone or support ticket regardless of your plan size and the pricing is simple with monthly caps that ensure no hidden fees and generous monthly transfer that's built in which means no large bill surprises like you get from aws or azure so go ahead and check out linode they are actually my infrastructure provider i love their service so go ahead and sign up with the link in the description below to get your 100 in linux server credits and now let's get back to the video [Applause] [Music] all right so let's go ahead and get ansible installed if you don't already have it installed real quick i'll just give you an idea of my setup here so you can understand the flow of this tutorial so what you are seeing on the screen right now is obviously a terminal but more specifically i am running tmux inside the terminal now tmux is beyond the scope of this video i actually have an entire series about tmux if you want to learn it but the reason why i mention it is because i have two tabs open here the one on the left which is highlighted at the bottom is my local laptop so what i'm going to do on the laptop is write the ansible configurations and push those changes up to github then over to the right on the second tab i have an ubuntu server on the ubuntu server i'm going to have it run the ansible configurations in this tutorial i'm only going to show one server now obviously you can have hundreds of servers with this method but i only need one to show you guys how this works so here on the laptop tab i am inside the ansible pool directory basically the repository directory that we pulled down so that's where i'm at right now and i am in the root of that directory we only have the readme file at this point now first of all we need ansible to be installed and if you haven't already done that it's just a matter of running sudo apt update and if you are using a different distribution on your workstation you can adjust that command accordingly but when it comes to debian or ubuntu or anything based on those distros the command should work out the same so we should be able to run sudo apt install ansible just like that now i already have it installed on my end so if i run this it's just going to tell me that it's already installed now when you install it you get the ansible pull command automatically i didn't do apt install ansible pull you just install ansible and you get ansible pull so that's pretty simple so next we can actually create our first ansible playbook and i'm going to start you guys off with something really simple so in my end i will use nano and i will edit a file by the name of local.yml and again i am in the root directory of the repository this file does not exist yet you can name this something else but i don't recommend it because ansible poll actually expects to find a file in the root of the repository named local.yml i'm not going to show you guys how to run a different playbook in this video it's actually pretty easy it's beyond the scope so i'm going to leave the name as exactlylocal.yml so i'll press enter and of course we have an empty file because it doesn't exist yet so what i'll do is paste in the contents right here so that you guys can see what i'm doing so here we have the first version of our local.yml file this is a really simple one and i'm going to go over it to make sure that you guys understand it now first of all this is a yaml file you do not need to know or master the yaml format for using ansible that's something that will come automatically as you learn ansible now yaml actually has other use cases and other technologies use this file format as well so this is not specific to ansible although the verbiage contained inside is essentially a yaml file is just a simple text file that contains a list of instructions and the instructions in this file are specific to ansible the first line here is where we call out which hosts we would like to run ansible against now normally that could be something like web servers database servers or whatever category you have created or role you have created that's beyond the scope of this video but since we are running this against localhost then we're going to include localhost here normally ansible uses ssh but since this is using ansible pull and we are running it locally we don't need ssh for that so we are setting the connection type to local so it doesn't default ssh now in this line we have become true which is essentially saying to ansible we need to use sudo for this become is actually more advanced than that sudo is just the default method that it uses for privilege escalation but if you don't actually tell it what kind of privilege escalation you would like to use it's going to use sudo down here we have a task section and the first task we have a name we're giving it a name of install htop now here we are using the apt module and this is for debian and ubuntu systems so if you try to run this on a centos system it won't work you can use dnf or yum for that so you might need to change that accordingly and then we are installing a package by the name of htop if you are using a different distribution other than debian or ubuntu i will leave it up to you to change the name of the package accordingly now when it comes to indentation it's very important to have everything indented properly the first line is all the way to the left and the two lines below it are tabbed two spaces to the right they are lined up with the h so here we have a hyphen which is like the beginning of a section and then down here we have tasks we have the name lined up directly underneath that two spaces over is apt and then two spaces over from there is the name so far that's our local.yml file and this is the file name that we are going to be working with throughout the rest of the video but essentially that is how the file is laid out we will be adding more to the file shortly but there you go that was our overview of the first version of the local.yml file so we should be good to go ahead and run it so let's go ahead and save the file we'll exit out and we'll do git status we see that we have the local.yml file here so we need to go ahead and push that up to get so git add and then that file then i will commit it with a message it doesn't really matter this isn't a tutorial on git but you get the idea press enter now that we have the commit staged we can go ahead and push it up to github just like that and there we go so up here i go ahead and refresh the page we now have the local.yml right here and what's pretty cool is that we also have color syntax highlighting in github as well because github understands what kind of file this is now back on the terminal let's go ahead and switch over to the server and we can go ahead and run it now first of all we will need to have ansible installed on every single server or workstation that we intend to configure with ansible pull now earlier in the video i showed you how to do that on the workstation on my laptop so to speak but i don't actually have ansible on the server yet so sudo apt update for debian and ubuntu systems then sudo apt install ansible enter that should be good to go we now have ansible pull at our disposal and now we can go ahead and run it so the way we do that is sudo ansible hyphen pull dash capital u and then the url now you might already have the repository url in your clipboard but if you do it's probably the ssh version what we actually need is the https version instead so back in our repository we can click the green button we can copy the https url then we can go ahead and paste it right here so to explain this command a bit we have sudo we actually need to authenticate with sudo first even though we have become in the playbook itself we can simplify this later and we will in this video since this command will make changes to the server we will need to use sudo and then we have ansible pull that's the topic of our video today no surprise there we have the dash capital u option which is short for url and then we have the https url of our repository so let's go ahead and run it and see if it works now that kind of went by really quickly here but we see that there has been a change and we also see some warnings here we can actually ignore these warnings and pretend that they don't exist essentially with normal ansible we would have an inventory file that would have a list of all the hosts that we intend to manage with ansible since we are using ansible pull we don't actually have an inventory file so we can ignore that message because everything as you know by now is being run locally now if i scroll up a bit you can see that i ran the command right here starting ansible pull so it started the job it pulled down the repository and you know we get another warning here about the host pattern we can ignore that again these warnings are just specifying that there isn't an inventory file which in our case is totally fine and then we see that it's executing plays against the localhost it's gathering facts which ansible does it's basically checking the server workstation whatever it is to find out what kind of operating system it has it's cpu it's basically taking an inventory of the machine itself and then here we have the install h-top task that we created in the file and it says changed localhost which means that it did install that package then down here we have two things that are okay basically two things that didn't need to change one thing did have a change right here that was the installation of h-top and then we're good to go so continuing on let's go ahead and expand our playbook to install more than one package and that's how this video is gonna go from here on out is we're going to be adding additional functionality to this playbook as the video goes on so let's go ahead and bring that back up so it's actually very easy to install more than one package with a single play so when i say play i'm referring to basically a single task so this play or task right here is just installing h top so what i can do is i can actually just change the verbiage so anyway i will go ahead and move h-top to its own line and i will go ahead and tab it over two spaces and put a hyphen there and we could add a few more packages here so on a new line just go all the way over here midnight commander that's a fun one to install and tmux as well so instead of installing just one package h top i'm installing a few more so that's essentially all we're doing here so i'll save the file then we can push our changes up to github so again get status we have that file modified the local.yml as we would expect we'll add it go ahead and create a commit easy enough so now that that is added we can go ahead and push it up to github and we should be good to go so now that we have that committed up to github we can go ahead and run it against the server and it says changed equals one we see that right here even though we have installed additional packages that was all in one play so it's only going to register one change for that entire play but we have midnight commander now and we also have tmux as well so it was successful so back here on the workstation we can go ahead and add the next change and expand our playbook even further so now what we can do is change the layout and improve the organization of our repository a bit here and i'm going to go ahead and demonstrate that so what i'm going to do is make a new directory called tasks just like that and what i'm going to do is move our local.yml file into that directory but with a different name and i'm going to call it packages.yml now the problem is we don't have a local.yml file anymore because we moved it into the task directory but we do need a local.yml file so what i'm going to do is create a brand new one and now it's an empty file well of course because we moved it into the directory with a different name so what i'll do is paste the new version in the file right now and here it is so first of all the first line hasn't changed the second line hasn't changed and neither has the third now previously this said tasks that was what it was called but we're doing pre-tasks and as i go over in my ansible series a pre-task is basically something that you want done before anything else and in this case it's going to use the apt module to run the update cache option here which is essentially the same as doing app update to synchronize the repository index that's what we're doing now here we have changed when false because we don't care if this changes that's almost always going to be the case so we'll ignore that now here at the end under tasks we just have one line here which is an include statement and it's going to include the packages.yml file which was our local.yml and that's in the task directory so it expects to find a task directory right here and a file name of packages.yml inside there which it will find because we already took care of that so i'll save the file and close out now before we run this we do want to make a quick change here so i'm going to bring up the original file that we renamed into the editor here and i'm going to press enter and we need to make a few changes here so what we're going to do is delete most of the lines here until we have just the task itself and i'm going to line everything up to the left two spaces to the left because now that we are including this in the new local.yml file we only actually need the task that we're running in this file right here so i'll save it and we should be good to go as long as i didn't make any errors here let's go ahead and push this up to github and i'm going to ignore all get etiquette and basically just do git add and then a period and that basically adds everything to the commit then i'll do git commit and a message of reorganized file layout i think that's good enough let's go ahead and push that up to github and there we go so as long as i didn't make any errors here we should be able to go ahead and run this and see if it works and it did work now nothing changed because we didn't actually make any changes i mean we did but the changes that we've made were only for organization we didn't actually add any new packages or anything like that so there was nothing to actually change on the server because the previous run has already added all the packages that this one installs back here on the workstation we can just keep it going and add some additional changes to our files here now what we're going to do is go into the task directory here we only have this one right now we're going to add a few more and this is where it gets really fun now first of all what i'm going to do is create a new file here called users i'll press enter so it is pretty cool that we can run one command to check in to the github repository for ansible and then run that ansible configuration locally but it would be much better if all of our servers checked into github automatically and we didn't have to tell them to do so so in this file right here this users file we can add any users that we want to add to all of our systems but what i'm going to use this for is to have it create a user for ansible to run ansible jobs in the background so what does that look like well i'll show you so what i'll do is go ahead and paste it in and here we have two plays that are being run so the first play here we have a name and this again as you probably already know is how it shows up in the log when this task is run we're going to use the user module and then we're going to use it to create a user named ansible and we have the system option we are setting to yes it defaults to no there's a bunch of options for this but we're only including the options that you know we want to change from the defaults which are only these two then we're going to set up a new task we're going to use the copy module to copy a file from a source to a destination now we haven't created this file yet and we don't even have a files directory yet we'll be creating that shortly but essentially what we're telling ansible to do is to take a file called sudoers underscore ansible inside the files directory and then put it in etsy slash sudoers.d slash ansible on the target we're going to set the owner of the file to root the group to root and the permissions to zero four four zero so easy enough we'll save it now what we could do is do mkdir files and then we could do nano files sudoers underscore ansible press enter and this file is only going to have one line which is this one right here now what this is this sudoers file is an easy way to add a sudoer to a machine now almost all linux distributions allow you to do that by adding a user to a group but you can also do so by adding a single file to the sudoers.d directory inside the etsy directory which is what we're doing here and this file right here we're basically setting up the ansible user to be able to run sudo with no password which is what is going to allow it to essentially run commands in the background without our assistance so we'll save that now next what we're going to do in this task folder is create yet another taskbook so we'll do nano cron.yml and press enter and here we have the actual code for this file as always we are setting a name and here's a brand new module to use called cron and if you are familiar with linux already you know what crown is but for those of you that don't cron allows you to schedule a task and we're going to run a scheduled task as the ansible user we are going to give the task a name of ansible provision we are going to run this task every 10 minutes and then for the job we are setting that equal to user bin ansible pool which is the fully qualified path for the ansible poll binary now we already know what dash u and then the repository url is we already know that but what's up with this dash lowercase o what does that do well what that does is it actually sets sensible pool to only run if there are actual changes to the repository so it's going to run this every 10 minutes and check to see if there are changes if there are no changes it's going to abort nothing to do no one has committed anything to the git repository so you know no need to run but if you do actually make any changes to the git repository then it'll see that the repository has changed that there has been a commit and it will run it so what that means is your servers will check in every 10 minutes and if you have already committed any changes since the last time that it's run well it's going to go ahead and run that so effectively you run ansible pull manually once and then what it's going to do is create the ansible user it's going to create the sudoers file for the ansible user and then it's going to add the cron job for all the subsequent runs so run it once you're done you never have to run it again all of your servers your laptop your desktops whatever you are configuring with ansible from that point forward will go ahead and take over from there now there is a way to make it send you a message if there is an error for example you might want to look into that otherwise you have to check manually that everything is running properly i'm not going to get into that in this video but just so you know that's something to look into later and then at the end here if i go all the way to the end it's wrapped a little bit it's basically sending all the output to devnull and you know that avoids any email messages or anything like that you could change this to redirect it into a log file if you'd like to see what's going on that's perfectly acceptable but anyway that should do it let's go ahead and save the file and close out now what we need to do is go back and we need to edit the local.yml file yet again the reason for that is because we are including the packages taskbook right here we are not including the new ones that we've set up so let's go ahead and add the new ones so tasks and then users so now we have the new taskbooks added to the main yaml file right here so we'll save this one close out and we should be good to go so let's go ahead and push our changes to github so what i'm going to do is cheat the system a little bit i'm going to add everything then git commit i think that's good enough let's go ahead and push our changes upstream and now we should be good to go back here on the server let's go ahead and run it and let's see if i've made any mistakes or anything like that is this going to work it's time to find out and no errors changed equals three that's what we see at the bottom let's go ahead and look at that so if i do an ls against the home directory we have the ansible user right there we can also see the ansible user added to the end we set that as a system user which means it has a uid of under 1000 so it got 997 for that so if we check out the cron tab for that user and we can see the cron job that ansible added which is this one right here and it gives it a name it sets it to run every 10 minutes and it's basically just pulling the url from github and then it's sending the output to devnull but it will run now again you can actually go ahead and redirect the output to a log file if you wish but that's all we needed to do to go ahead and get this automated so that all the future runs of ansible will be automatic so now on all your other servers workstations whatever you have you can basically run the ansible pull command manually then all of those machines will check in automatically from that point forward and you can go ahead and start pushing your changes to get and then all of your machines should obey your every command check into git download the changes and then run them so there you go that was a lot of fun i really enjoyed that you know i have a lot of fun with ansible in general it's one of my favorite technologies i can't say that enough i had a lot of fun doing the ansible tutorial series that i just recently wrapped that i highly recommend you check out if you want to learn more but even if all the ansible you do is the ansible pool tutorial that this video helped you out with then you know it's still a lot of fun because with what i showed you in this video you can use that to centralize all of your configuration management to get have all of your servers obey it's just a really awesome way to go ahead and do that so what do you think let me know your thoughts in the comments down below thank you so much for checking out this video subscribe if you haven't already done so and i will see you in the next video you

Channel: Learn Linux TV

Views: 22,730

Keywords: LearnLinux, Linux, Tutorial, Review, Howto, Guide, Distribution, Distro, Learn Linux, operating system, os, open-source, open source, gnu/linux, git, github, devops, automation, ansible, ansible pull, ansible-pull, pull, gitops, ansible tutorial for beginners, ansible tutorial, ansible playbook tutorial, ansible playbook, ubuntu, debian, centos, script, scripting, automate, ansible git, gitlab, version control, ansible automation, open source software, linux commands

Id: sn1HQq_GFNE

Length: 40min 55sec (2455 seconds)

Published: Thu Sep 10 2020