Ansible 101 - Episode 7 - Molecule Testing and Linting and Ansible Galaxy

good morning everybody or a good evening good afternoon whatever time of the wool whatever time it is in the place that you are as with all these live streams it would be awesome if at the beginning or whenever you see this you would put where you're from it's good to see people from all over the world come together for these and learn to automate things for together and and do new things you might notice today I am wearing a volleyball shirt I don't know if you enjoy the sport of volleyball or not that it was my favorite sport in high school and I'm wearing this shirt in honor of all of the high schoolers especially but anybody else who has missed out on their their spring season of sports or music or arts or whatever kind of activities that they did and didn't get the opportunity opportunity to do this year I know for myself volleyball was one of the things that brought me back from from not quite depression but I had some major issues with Crohn's disease I have a chronic illness called Crohn's disease and I got that in high school right before my volleyball season it kind of knocked me down a few pegs and took me from being a very very good volleyball player to being a mediocre at best volleyball player I went for anybody who knows volleyball I went from being an outside hitter to being a libero which is kind of like if you were in soccer being a field player to being the waterboy in a sense but it was it was something that really gave in and gave me the opportunity and willpower to kind of fight back and get my strength back after having a pretty rough go with that disease and still something I struggle with every day but I know that that was a big thing in my life and it it's difficult seeing people not be able to have that that year of their lives especially for volleyball for me but for anything whatever it is in other young people's lives so please always reach out to people try to help anybody who might still be struggling even as some parts of the world are opening back up and having people go back to work and go back to different activities there's a lot we still can't do and you still need to be there for other people today's episode is I'm excited about it because it's something that people have been asking about forever I've had an issue open on my books repository forever talking about molecule testing and we'll get to that as with news casts that is the big topic but it will be at the end I am disappointed to say but it we will have time to talk about it and depending on how much we get done today I might make this a two-parter and have it also in episode 8 we'll see but I wanted to start off by thanking especially the new github sponsors and patreon patrons who make this possible and who make it possible for me to spend more time doing open source work Matt Gleeman Dave runs co uk I'm guessing from the UK matthew cosgrove and mark brennan and deanna p on patreon they have started supporting me this week and actually this week I've I've added my support for a couple other people on patreon to whose work that I use often and I really encourage you especially in times like this some of the people who do these open source things or contribute content for free they rely not entirely usually but a lot on donations to be able to spend time doing these things and having passion projects that you might benefit from so please consider getting back whether to me or other people that you rely on for their work and it's great to see everybody here it looks like there's people from Des Moines Iowa Malvern England UK New York Sweden Greece Denmark Morocco Virginia London Florida Austria so awesome to see everybody here glad you could make it and if you haven't seen past episodes please feel free to go back not during the live stream because you want to stay here with us now but if you can go back and see all the previous episodes one to six of the series on my youtube channel another thing that I've been starting lately and I have the second video almost ready to go is a series on a Raspberry Pi cluster how to build a Raspberry Pi cluster and in this case using the Turing PI this is a prototype board one of the kind that I got from the Turing machines company that is making this and they're letting me test it out and I wanted to do another video series on how to build clusters with raspberry pies and specifically a kubernetes cluster and the next episode I'll be discussing how to put together the hardware and then another episode I'll be talking about installing k3s on it so if you're interested in that click Subscribe below me you can see all the new videos that I put out on the YouTube channel and also get notified when these live episodes come up there were a few questions and and statements from last week's episode and from a couple other episodes that I wanted to highlight one was from Baptiste 8mm any hints on when a team works on the same playbook is there a way to avoid password sharing this was talking about ansible vault there's there is an option called vault ID that lets you have multiple passwords sign one vault and you can have different passwords for it so if you're interested in that using vault with a team that is one way to do it rolling pictures asked dumb question this is not a dumb question this is an excellent question do we need configuration management tools like ansible if we're working in docker that is a very your you'd be surprised that's actually a very good question because there's a lot of different ways to approach that question and I would argue yes because docker is one of many different tools in a full deployment and production pipeline for applications and it depends on where you're going with kubernetes with running docker compose using backer swarm or other tools out there for running the infrastructure and also testing things and building things so that's something that I'm not going to cover in a short answer here because I can't but I do have a blog post on an Sable's blog the inside playbook that talks about whether ansible is necessary in a cloud native environment so I encourage you to go check that out you can search Google for it and I also have a book called ansible for kubernetes that begins with docker so and then also in Chapter 13 which I think it's yeah chapter 13 which soon will be chapter 14 actually in ansible for devops has a whole section on dock that you can you can explore and how ansible can help with your docker workflows Patrick Cole asks and I probably butchered that name should I structure the play books as roles or would you recommend doing collections that's another thing that I'm not going to get into today I'm actually reworking some of the content in the book to discuss collections but it's not something that's really concrete yet and so I don't want to write on a shifty foundation I want to make sure I have enough time to understand all the different areas and collections that could benefit organizing play books that way so I would say for now I would stick with roles and I'm keeping the books content in this ansible 101 series focused on using roles and play books just because they're simpler they're easier to setup they're easier to maintain and manage and we'll discuss ansible galaxy a tiny bit today but that has some implications too for roles versus collections Stormo fo.1 has been posting some comments on my videos with tips for beginners which is awesome please feel free to add comments on these videos with anything that you learn that that it would be beneficial to other people watching them in the future but he mentioned that if you don't have if you don't have an ec2 instance available or if you don't have an account on digital ocean or something like that you can run VMs with vagrants any of the examples that I have they're meant to run on any server it doesn't have to be on Amazon it doesn't have to be on vagrant if it's if I test with vagrant same thing in the book a lot of the examples might use vagrant but you can use those play books anywhere as long as you have the same kind of server so if it's a bunt to having a bunt to server running whether it's a VM whether it's a docker container in some cases or if it's as easy to instance or even a bare-metal server you can have a server running at your house you could even have a Raspberry Pi zero running using raspbian and you can target that with playbooks to an errand Colby and I might have butchered that I don't know how long are you going to continue live stirring pass device 42 sponsorship well I have to say thanks again to device 42 but their sponsorship is over the books are no longer free for now and I plan on continuing this as as I can I just don't know I know I have plenty of content so I know one episode I'm going to talk about securing servers like the first five minutes on a server with ansible automating the security I have a chapter on Windows and ansible I might even pull out my Windows laptop and stream straight from it and see how that how spectacularly that could fail ansible tower and awx ansible and docker ansible and kubernetes and lots of different things that we can talk about so there's definitely no lack of content I think at some point I might have a pause after I cover most of the main parts of my book and then I might do a new series after that focused on a specific topic but we'll see so those are questions and answers and Devon in live chat is also asking about deploying awx on kubernetes it's funny you ask that I actually am interested in the same thing and that's why I have the tower operator which is right here which deploys awx on OCP or kubernetes OpenShift and you might be interested that it is not supported it is not in an official installer so don't don't take it to production necessarily but it is the quickest way that I use to get ansible tower and native FX running but we're not talking about that enough in this episode so I'm not going to go too deep into that we are talking about this episode ansible galaxy and before I go too deep into that I do want to mention for many of these things I have content outside of this livestream series and even outside the book that talks about some of these topics more in depth for example I gave a presentation at ansible fest in 2019 in Atlanta that was last year back when we could meet together in person and it's called there's a role for that how to evaluate community roles for your playbook and I mentioned last week that you had play books like here's a play book that it's a little different than last week but I had a play book that used Java and solar to roles that I maintain and lets you build us a solar server that that is built for search very quickly and easily using ansible community galaxies rolls and this presentation kind of goes through how do you evaluate roles and see if they're good for you or if you might not want to use that role if they're well maintained all those kind of things so you can go into there for more detail but I did want to talk a little bit about ansible Galaxy itself and how how you can get rolls into your playbooks from ansible galaxy because I just showed you how to use it but I didn't show you how to actually get the roll to your computer and the way that I recommend there are many different ways to get ansible Galaxy rolls locally the easiest way is just use ansible Galaxy install actually ansible Galaxy roll install and then give it a roll name that's on ansible galaxy so this is if I go to ansible galaxy galaxy ansible comm and search for homebrew i should find my roll summer up here here it is so any roll on galaxy can be installed with this command but if you just blindly run this command it usually installs it into a global location like on my computer I think it would install into / at c / danceable / rolls so I don't want to do that because I might have different versions of the roll for different play books and I'll talk about why I do that in a second but I like to install my rolls local to the playbook so in the same directory as my playbook so in this case what I'm gonna do is I always create an ansible dot CFG file alongside a playbook and in here I give a rolls path in this file should you have an extra line at the end so that it doesn't complain but whoops the rolls path tells ansible where to install and where to find rolls so right now you can see there's no rolls folder in here but the rolls path is set to that so if I run this playbook with these two rolls over here in scible galaxy or ansible playbook and this is what I mean Hamill and I use it with - Kay because I need to enter my become password on my Mac it's not going to find that that role because it's not found in the rolls this rolls past can be multiple paths you can chain paths together if you want but for playbooks I typically like to only have it look for roles inside that play book directory because if you look in a global location the version of a role globally might be different than what you need for a particular play book and that can screw things up so what I want to do now is make sure that these roles are installed and so you can add a requirements file requirements camel that lists all the roles that your PlayBook uses and just like Python requirements or just like a composer JSON file in PHP or package info I forget what it's called for NPM node yeah somebody can throw it into Live Chat just like those those list files requirements lists a list of roles that you want to install this can also list collections which we aren't covering today but you all you have to do in the requirements file is give a list of roles with a name for each role you can also provide a version if you want to lock it in at a certain version so let's say I want to find this one his command-line tools are here on galaxy and the latest version is two point 3.0 so I can say version whoops version 2.3 point oh and for my homebrew the latest version of that is maybe just paste it in here and go to the URL the latest version of this is 3.1 point I'll save that file and then in my terminal I can run ansible Galaxy install - R - R says pass at a requirements file requirements animal and what that's going to do is download these two roles into a roles directory right here for me and now I have those available and I can run my playbook like this providing my password so I ain't smokin stored in memory and now it's running my playbook which is configuring my local Mac and making sure that pv is installed via homebrew and this is an extremely simple example of that Mac dev PlayBook that I maintain that maintains all the stuff on my confuse and I highly recommend it if you use more than one computer it's nice to have an instable playbook that manages everything on that computer and if you're if you're interested in how that works Mac dev PlayBook is on github right here and has a fancy logo with a little ansible inside of a laptop but people can do the same very that the exact same thing on a Linux machine you could probably do it on Windows too but it might be a little bit more difficult to run from the same machine on Windows there's on a Mac or Linux know so that's galaxy I just wanted to point that out and then also recommend this presentation there's a role for that and I'll paste a link in the chat so that you can grab it there since you can't click on a link through my screen and you can go there and watch that episode or watch that that presentation later I have the slides for it also up on my website Jeff Garlin Netcom so after the video is over I might throw a link to that up in the description if I remember so anyway that's ansible galaxy there's a lot more to cover on it and I think I might do an episode on collections in galaxy at some point in this livestream series and at that point I'll cover galaxy a little bit more in depth but I really wanted to get into testing in scible and and talk about that today so the first example I have is well let me actually get started so galaxy was in Chapter 6 of my book this is the book ansible for devops you can get it and ansible for de vacas calm and there's actually a link in the description right below but I was talking about galaxy from chapter 6 and we're actually skipping a few chapters there's chapters on kind of ansible cookbook some different things you can do with ansible to deploy applications and I probably won't take one of those and do it use it as an example in one of the episodes coming up and Devin says we'll remind me about about putting that Lincoln thank you very much if you put a comment in in this video I'll definitely remember we're skipping a couple chapters also a chapter on security I'm going to get to that later but I skipped ahead to chapter 11 which in the current version of the book version 1.2 - is automating your automation ansible tower with CIC CD and originally I had a very short section on tower and then a long section on using Travis CI for testing it rolls but I when I wrote that chapter originally ansible lint didn't exist yet molecule didn't exist yet I hadn't been using emmalin and all of my testing was basically running a role in a docker container and then re running it and making sure it worked so since I wrote that chapter so much has changed and the way that I do my testing has changed completely so the chapters a bit out of date so you are seeing on this live stream the first public revelation of the new chapter 12 which is bumping chapter the existing chapter 12 and 13 and 14 up one chapter and I'm going to rewrite chapter 11 to cover ansible tower and awx in more detail in chapter 12 is going to be all about testing and the vitaliy is asking about how to use molecule with github actions we're gonna get there don't worry but we're gonna start with what I call the instable testing spectrum and also discuss a little bit about what I call what I would call unit integration and functional testing in ansible itself so with any testing it's important to have different different layers of testing usually on the code level so if I go to if I go to this playbook that I have right here I mean that amyl the first level of testing for ansible content is does is this valid animal so we want to we want to check that a second level would be is this valid ansible yeah Mille is this a valid playbook a third level would be does this ansible playbook run in a fresh environment and a fourth level might be does this the sensible playbook run item potently and will and run against production multiple times without breaking things and then another level that most people don't necessarily need but they might want is to have a parallel environment that's basically the same thing as production so a staging environment or pre prod or something like that and rebuild that environment from scratch every time you make a change to make sure that everything it works correctly or you can even have Bluegreen to production employments and if you use something like kubernetes that's that's even easier nowadays to do but I like to call this the ansible testing spectrum and I have this beautiful graphic over on the side here with the rainbow I have no idea where I got that from but this is a slide from a presentation I gave him 20 18 on maintainable ansible playbooks and I mentioned playbooks because that is the unit of automation that I think the the most effort for testing should go into when you're building things it's important to test rolls individually that's kind of like unit testing in code but but at the playbook level you really need to make sure that you have that you have that that vast testing level that is going to make sure you don't blow things up and it starts at the top this is the easiest thing to do and gives you a good bang for the buck but maybe not necessarily the most bang for the buck is Yemma lint which makes sure your Yemma file is correct the next thing is ansible playbook with syntax check make sure that the basic components of syntax work with ansible and ansible can compile the playbook that doesn't necessarily mean it's going to run but at least it can can see everything and and that not blow up then there's ansible lint which tests for the yeah Mille that's in your playbook to make sure it's compatible with best practices and yet so ansible again it doesn't mean that it's going to run it just means that it's good then there's molecule test which is an integration test that runs your PlayBook against a fresh environment that we'll get into then there's ansible PlayBook in check mode which you can run against your production for structure to see am I going to break anything with a change or something like that and finally there's that last step which is way more complex and I rarely see this in practice unless you have infrastructure that has multiple millions of dollars writing on it which is to build parallel infrastructure tests everything on it and then tear that infrastructure down and then deploy to production so this is the different ways that you can test ansible and I notice thank you so much guru I can't pronounce your name gooo I'll call you Guru Prasad thank you so much for the free books in the video series you're very welcome and and I also I noticed a few people were asking questions in the live chat on different things and one really cool thing about doing these live streams is in the live chat other people answer the questions other people ask which is great because there are some people in this live chat believe it or not that know a lot more than me about ansible but they as with all of us I think we all have stuff to learn every day and and if you don't learn something new every day that then you're either fooling yourself or you need to talk to more people and learn a little bit more anyway thank you for the people who answer things in life streams too because I can't always monitor the live chat but if you do want to get me to notice your comment really quickly you can hit that little I don't I don't know what the button is there's a money button there that'll definitely flag my attention because it pops something up on my other screen anyway so these are the different levels of ansible testing and I'm just gonna walk through each one and you'll notice that each one is simpler than the others and one of the things that I do in all these layers of testing is make sure that I also have tests built in the ansible so in this new new rewritten chapter I have some text here talking about the best testing really and and the most important thing is to make sure that your ansible playbook itself does testing on the fly so you can make sure that you're doing the right thing in the playbook itself and even if you don't do anything else you can make sure that a web service is responding before you continue or you can make sure that a port is open or you can make sure that a file exists all those kind of things you can do that in line and in ansible playbook using ansible syntax which is really easy you don't have to have a separate tool that's written in Ruby or a separate Python thing running that you have to know a special syntax to use so the first layer of that is a lot of times when I'm building a playbook I just use the debug module and the debug module just prints things to the screen and the debug module also has a setting for verbosity so you can have it in here and it won't always print out unless you add more verbosity to Ansel's verbosity verbosity i don't know how to pronounce that but anyway it adds more if you if you increase for both city or verbosity then then it'll print the debug message if it's default then it won't but in this case it'll always print this debug message here and I've used this a couple times in live streams in the past CEO one instable playbook what is this debug ml and you'll see that it grabs it grabs uptime registers the variable and then it prints it out and and you can also use win clauses with debug to to show debug messages at certain points and not at other points pretty simple we've done this before on this live stream series and but it is sometimes you forget like oh I just need to throw this value on the screen so that I can check it later the debug is the simplest way to do that you can also increase Sansa Bowls verbosity to see the output of commands and things but debug can be helpful if you need to get the value of something or see what a variable structure is that kind of thing another pattern that that I use a lot is using the fail and assert modules so I'm gonna CD - OH - and then this playbook shows the fail model and the assert module and they're very similar in what they do it's just a slightly different structure to it so fail is always going to fail when it gets run so if you use fail and win together you can make your PlayBook fail at a certain point if a certain thing is triggered so in the first case I have should fail via fail true and so that's going to trigger this and the fail module is going to just fail the playbook at that point so if I say ansible playbook fail assert then it's going to fail right here an epic failure as it were but if I tell that to be false and say oops and say that it should fail via cert then the assert module is going to assert that this this variable should not should not equal true and if it is true then it's going to fail so it's kind of like the inverse of fail an inverse of fail plus win so in this case it fails here and it gives you the reason the assertion failed and it gives you the assertion so you can see in a playbook which assertion that you had failed so you can see how to fix that so I'm gonna set that to false and pop down here and say this true whoops I can't spell today there we go and this assert uses multiple conditions so again this is it's it's kind of the same thing as fail plus when it just depends on how your brains working that day I use both all the time assert is probably a little bit more formal and it probably should be used more for asserting different things are happening in your infrastructure but you could have something like a command that returns output and it checks that that something is in it so you can have assert that this thing the string is in this output or you could assert that this port is open or assert that there's this content in a web response different things like that and you can do that in line in your PlayBook to make sure that before the playbook progresses further something that you expect to happen happened so that's testing in line and you're in your playbook now getting to the actual test spectrum so once you have a playbook that's running there are some things that are really nice to do and this is a little confusing for me because I accidentally printed this double-sided and these pages are all like it's upside-down so I will be fumbling around a little bit with these pages since they're not in a nice book format yet because I can't magically print books at my home during during this time let me also I need this switch my monitor over here too because I can't see okay so the next thing is yema lint and Yemma lint emmalin will quickly help you weed out any yamo issues and so one reason I like to use this is because it's it's often the case that you have a playbook that runs but you have some spacing issues or other little issues that can lead to problems in the future because you might you might go to a level in the playbook and press ENTER and then your code editor puts you in the wrong tab level or something like that so it's important to have spacing correct in ya Mille just like with Python and if you have the spacing incorrect that can lead to problems in the future so ya M'Lynn is basically giving a set of guidelines for animal files that are nice to have and applying them to your to your actual playbook so this playbook here I'm gonna run it and it should work I believe so if I say CD 3 gamma lint ansible playbook lint example it still runs so it's a working playbook but there are a few issues with it and the egalité among you may have already found some of those issues if you want to feel really smart and stuff you can stick that in the live chat and and show everybody how you know everything about a moment without even running it Thank You Ross thank you very much someone else hit that little dollar button and see I've see it right away because it pops up on that other screen so anyway to get Yamma lint it's just like with all the other Python tools you use around ansible you use pip to install it so I'm gonna say pip 3 install a moment and hopefully this doesn't blow up my computer I know for my Drupal live streams github has been down two times out of the 13 live streams I've done so it's always risky doing things over the Internet but am a lint is installed and now I can just run Emma lint and then give it this current directory and it'll give us some of these things here and so a lot of these things are just like documents start is probably not that important but I still like being more formal about it because you know this is a Hamel document if you give that that three line start so we'll do that and fix that it noticed there's a trailing space which in my code editor is easy to see because I use a plug-in called trailing spaces to highlight those we can get rid of that that makes things a little more clean to few spaces before comment this is just a code style thing there should be at least two spaces before comments in the ML documents that makes it a little more that it's a comment especially when you're in a system that doesn't highlight the comments like my my coded others doing also this is something that I see all too often when I'm linting other people's ansible playbooks a lot of times they if somebody uses the space bar to make tabs like this which is something that in Silicon Valley annoys me to no end because people who use spaces for tabs don't use the space bar they don't go space space space space space like that they use tab and they have a code editor that has a tab with setting anyway I digress I am a spaces over tabs person I'm also of em over Emacs person and they probably just lost half the audience but anyways so that it found that there was a space missing right here I put that in and but there's one value here that in the Drupal and there in that Drupal I am confusing myself since I mentioned it about the other other livestream I'm doing in the ansible ecosystem a lot of people use yes or no or even on or off but mostly yes or no for play books for boolean values and yeah Milland that's perfectly valid you can see my code editor editor even highlights it as a boolean here and but yeah melons default rules don't account for that so it's nice to be able to tell and gamma lint and let me flip over my page here so I can actually see it it's nice to be able to tell yeah moment hey it's okay to use yes and no but I still do want to make sure that boolean czar a certain type of value yes/no true/false and not just anything under the Sun because technically I think you can also use 0 or 1 but that can also be challenged when you're dealing with integers versus boolean so I like to limit it to false true yes or no basically and we can tell Emma lint that using a yema lint file so first I fix those other issues that surround the a moment again and this is the only issue it's picking up now so I'm going to create a Yemma lint file so I'll save this as dot Amal lint I used that yes in this yellow lint file I'm gonna put extends default that means that it's going to use the default rules that Amal in ship with but I'm going to add in a setting for the truth a rule and you notice that this is y Amal so you can use Amal to configure Y a moment it's always fun like that loud values and then I can give it a list of true oops false which are the ones that it ships with and then I'm also gonna add in yes and no and save that file and now if I run it again it gives me no errors and if I say echo the RC it's going to be 0 so in CI if you have your CI environment set up to run Y Emma lint on your PlayBook it'll return an error if there's any if there's any errors or warnings in the Emma lint but it'll return 0 and pass if there are no problems so I usually throw a link Amal int and CI pretty quick because it's it's a quick and easy way to make your your playbooks all look pretty uniform and have all the right amal things like the dashes to start it the right spacing in the right indentation so and the next step beyond that is that that go for the next step beyond that is to do an ansible syntax check and this is pretty it's it's a pretty easy free way to get another level of testing done for your playbooks and cin it's super quick it takes maybe a second or two at most to run you in for a pretty complex playbook and what this will do is it'll check if ansible can basically put everything together like when you run an answerable playbook it has to put together all the imported tasks and playbooks it has to make sure that all the modules are findable it makes sure that variables are formatted correctly for ansible so in addition to being valid y amal is it a valid basic ansible playbook and if i run that here in scible playbook what is this called syntax check syntax check like this with a flag it finds quickly Oh oops I don't have a free animal file so it's trying to import this file but it doesn't exist so okay I'll fix that and I'll run this again and it looks like I'm good right unfortunately syntax check is not super intelligent because it's not actually running the playbook some things like dynamic includes so if you use include tasks instead of import tasks that's a dynamic operation that that happens at the runtime of the playbook so syntax check can actually check some of those types of things or if you have dynamic variables that use set fact to set them that kind of thing it can't be a hundred percent correct so that you know this should actually be date amel because if I do it like this and run the play book without syntax check it's going to fail right here even though the syntax check passed but if I do it with date then I can run it and it'll pass so syntax check is just it's nice to give a basic level of can ansible compile this stuff together it's not much beyond that it's not checking if the playbook actually runs or if everything that's listed in the playbook can be accessed so just keep that in mind it's not a golden panacea for for your testing let me figure out what paid them on in here and try to get to the next one with all these flippy pages so the next thing I want to talk about is ansible int so there's a moment for the amel basic Hamel stuff ansible int can also test ansible tasks and play books to make sure that that there's kind of the best practices for formatting tasks and writing play books are followed there's it's not a full breadth of everything in ansible but there are a lot of things that that can help you avoid pitfalls or avoid bad practices like using the shell module instead the command module so we'll go over to that example ansible lint oh five ansible lint and if I run this playbook it does work so ansible playbook mean that yeah Mille it works correctly and gives me the output I want I can also run a yam a lint on it and emmalin oh Emily did found one error so let me pop that in there so yeah moment passes and if I run the PlayBook it works but there are some things in here that I can do a little bit better so first of all I'm going to install ansible int which is just like all these other tools you install it using pips I'm going to say pip 3 install the instable int and I just realized I have my mail open I should probably quit that and now ansible it's installed so I'm going to run ansible int on it as well so ansible lint and then this is main that Gamal one quick note on running ansible in you can pass it a playbook or a role specifically or if you're in a git repository you can have it auto-detect play books and things but for for best for for best compatibility just pass it the play book that you want to check so I ran it on here and there are three different things that found in in the play book that could be correct it could be more correct so one thing is this task is not named so why am i doing this I you know in my case I'm doing this just to demonstrate a playbook running on my local machine but it's not obvious because there's no documentation for it so it is correct I should put name get system uptime to demo how this works so I fixed that issue which is right here 502 all tasks should be named another thing here is saying use shell only when shell functionality is required this is just running a command it's not using pipes like it's not typing that through something and then using awk or anything like that it's also not using and and it's not you know it's not doing anything that this that the shell module is really required for so it's better to use command to do those kind of things so I'm going to switch that to command and then also this 301 says commands should not change things if nothing is doing that's it's kind of a funny way to say that but basically ansible will always report this as changed and I know that when I run the uptime command I'm not changing anything on the system so I can put here changed when false and now if I run this playbook again or if I run ansible lint again it won't find any issues sometimes so there's a list here ansible lint what is it rules there's a list and the documentation I think it's under here of all these different rules it doesn't have it here but there is documentation rinsable and here's the default rules a few of these rules are a little bit more strict than I think I care about what is it five Oh what is it where is it there's something about like changed let's see commands ah so 503 I often ignore 503 because sometimes you do have a playbook where you want to do something and if that thing results in a change do something else immediately and this has a rule in here that says if you use changed in a win clause so if I say like when when system time that changed I believe that this is going or is changed I could use two if I do that it's going to pop this up so you know a lot of times it is better to use a handler for something that happens after something changes but if I needed to happen immediately it's nice to just be able to do it immediately and so you can ignore that using the the ignore syntax or you can also create an ansible lint file to ignore something like that but just something to keep in mind ansible lint doesn't mean if you if you don't do it the way that ansible lint wants you to do it you're bad in some cases it's perfectly okay to do that and that's all these lint tools they give you suggestions it's not concrete you have to do it but but these these three tools Yemma lint syntax check and ansible and are all great to throw into any CI for any playbook just to make sure you have a uniform set of rules and set of guidelines for writing your your syntax and that also helps other developers who might not be as familiar with the amal if you're on a team those other developers will have to kind of pass that gate to make sure that their code lives up to the same standards and is formatted similarly otherwise you end up with play books that look like this and over time you know this isn't a huge deal because it works but over time what happens is you start getting the wrong indentation level and it gets harder to figure out where you are in the PlayBook and what you're doing and and it gets really messy so using these organizational tools that to lint and check your code is really helpful now we've gotten through those things those are on if I go back over to my to my ansible testing spectrum those are all in the top three and the bottom three are where we're kind of rubber meets the road where things start getting things start getting a lot more useful but also a bit more difficult and molecule I wouldn't call it difficult necessarily I would just say that it is very broad and you can do a lot of things with it and because of that the documentation can be extremely daunting I know the first thing that a lot of people do is they go to molecules documentation and they they glanced at something like the getting started guide and some of these things it's just a lot to see in the first run-through and I know that the people who maintain molecule the people who are few and proud and do a lot of great things they have spent a lot of time trying to get these back Docs a little bit better but I think in the past especially it was even more daunting and challenging because the documentation wasn't organized in a way that that was really targeted it's somebody brand new to it so that's one of the goals of this videos to help dispel that myth like molecule is actually a very simple tool and the cool thing is so let me actually get to my section and I my pre-production first ever seen a copy of chapter 12 basically one of the things that I used to do when I was testing play books and developing them and trying to build them the first time is I would build a VM somewhere whether it's in vagrant and VirtualBox on my local computer or on Amazon ec2 or something like that then I had set up SSH so I can connect to it I'd create an inventory file so that my ansible playbook can connect to the VM and also if I had a production playbook I'd make sure that it was not connecting to production that can really trip you up if you accidentally do that and then I had run the playbook against the VM I'd test and validate things I'd do some development work and then at the end of it I would delete the VM and it's a really heavyweight operation especially if you maintain a lot of different things like I do but even if you only maintain a few different ansible playbooks it's a lot of work just to you know it might take you five or ten minutes to just get set up or thirty minutes or an hour if you forgot your login or you got to get your two-factor password for Amazon to get your VM running all these different things and vagrant helps a bit so in some of the examples in the book we've used vagrant to build a VM and run an answerable playbook on it and that's nice but it still is a little bit heavyweight and so that's that at that point in my time using ansible I started writing a shell script and the shell script I believe is still out there if I go to just that github.com and search for ansible tests or something like that well I'm not gonna find it that quickly but I have a just out there that has the script and it's kind of beastly and an added support for more operating systems and things it worked and it was great but it didn't have a lot of plug ability it wasn't extremely maintainable and it was a shell script and it's like I'm writing all this automation for my infrastructure that my testing is based on a shell script so right around that time was when molecule started to become more than just like a little side project for someone and got a lot of uptake and I decided to start moving all of my ansible roles to use it and then I realized it's it's not great just for role testing it's great for playbook testing and I also use it now for testing my ansible operators and kubernetes molecule can plug right into Cubert kubernetes running a local cluster and testing things in there it's basically a way to use ansible to set up a test environment to run an ansible playbook and then do tests on the ansible environment and then tear it down all using ansible it uses some Python of course on the back end because it's written in Python but but it's it's mostly ansible and it's pretty approachable from that perspective and to get it you just need to run let me go into that folder oh six you'll pip3 install molecule you just install it with pips reinstall molecule and there's a few different ways to use molecule so it was initially built with just ansible roles in mind but it can be used for a lot more than that today since I don't have the time to get deep into all the ways to use molecule I think I might break that out into another session next week right I show how I use it in some of my other projects but roles are the kind of the simplest and easiest way to demonstrate how quick it is to get started with molecule so I'm going to use molecules built in command which is molecule in it roll and then give it a roll name and this is the exact same thing as running galaxy or ansible galaxy and it roll in it I think my role so if I do that ansible galaxy will create my role it has the typical ansible galaxy role structure it includes all the different things we talked about last week and but in the end what's really required for a role is just tasks and a meta / main amyl with dependencies listed and it gives you this role structure I'm going to delete that my role and I'm going to use molecules command which is the exact same but with molecule some molecule role in it my role and that's going to create the exact same role structure in it role okay that's something we might want to change in molecule itself and that role there we go because molecule like I said it was built just for roles but now we have collections too so at some point we'll probably have molecule do collection and it's as well anyway so it created this role and it's the exact same as the role that we created with galaxy however there's one minor difference you might notice that there's a molecule directory and this is all that's required to test with molecule you just have to have a molecule directory and inside that directory you have one or more scenarios molecule ships with one default scenario but you can have like you can have a scenario that tests in one condition that tests another for example with my tower operator that I mentioned this beginning of this video I test it in one scenario tests using a kind cluster which is local docker based kubernetes and docker cluster but there's another scenario that can run it on on mini cube which is a different way of running a local kubernetes cluster so you can have different scenarios for different environments or you can have different scenarios for different types of use of your PlayBook or role however you want to do that is is up to you in the most in most cases I just have one scenario which is like the default way to use this role and then I might have a playbook that can go different ways depending on environment variables something like that but it gives you one default scenario and it gives you four files in here this file just shows you how to set things up so that you can run so that you can run molecule but we already did that by installing molecule then there's a molecule ml file this describes how molecule is going to run tests and at a very basic level it gives you a few different things one is dependency dependency management this is always going to be almost always going to be ansible galaxy at least in all my examples and then it uses a driver for running the tests so in this case we're gonna use docker and that's the default it's going to create a docker instance locally and it's going to run the run our tests inside that instance and this is going to delete it and that's you can also find drivers for VirtualBox and Amazon ec2 and other ones so that you can build instances there and run the tests in them and then it gives you it gives you a little bit of control over how it runs inside docker and next episode I'll get into a lot more detail on how you can do more advanced things with this but by default it's going to run a sent to us container and let you test inside of it obviously if you're building for a bunt to you're not gonna want to use sent to us so it would be good to cover how you can test in different platforms and different distribution versions which I'll cover next episode then it gives you a provisioner this is typically going to be ansible because we're testing ansible stuff that's that's pretty simple and then finally a verifier that's going to run after everything is set up so that you can test that everything's working correctly and by default that's also ansible so you can run a playbook to test your ansible playbook in some cases that might be redundant if your playbook already does all the testing that you care about making sure that things are running correctly and that's great but a lot of people also want to run extra tests to kind of check that the whole system is working the way that they expect afterwards and so the other two files go along with this converged AML is a playbook it's a very simple playbook and all it does is run the role that molecule just set up for you so this is my role and this playbook is going to run my role that's all and this this by the way is equivalent to running roles my role so we could delete that and it's a super simple playbook the the main thing is it's called converge that animal because molecule converges everything together through this playbook so if you had other setup steps and to get your role working you can run them here so you know if you're if you're on a bunt to a lot of times you might need to make sure that the apt cache is updated so you could say apt update cache equals yes and and then give a cache lifetime so you can change this playbook to do whatever and malloc in molecule will run that run this playbook whenever it sets up your environment then the verify playbook lets you test things inside of that that environment so this is just a certain that true is true which is pretty pointless but we'll get we'll see how far we can get with testing an actual real-world case here but just to demonstrate I'll save all these defaults and we can run molecule test and that's going to do all the different steps which did I change something in here I didn't molecule glob failing oh I'm not in I'm not actually in the directory so I need the CD into my role and then run molecule test and molecule is going to find that that configuration and it starts doing different things I'll go back here so it's showing you the full matrix of all the things that's going to try to do some of these things that need to do because it doesn't have any ansible galaxy dependencies to install so you can see it's skipping that missing that file now there's no cleanup playbook that we have custom typically I don't have this for my rolls linting is disabled because there's no linting configuration here all these different things but what it does here is it hits first it makes sure that the entrance instance doesn't exist because it wants a clean environment then it creates the environment this is building that that docker environment right here creating the molecule instance again this is all using ansible so it's kind of nice in that regard that you use ansible to build the environment that you're going to run your PlayBook in and then use ansible to tear it down then there's no prepare PlayBook so it skips that then it hits converge where it runs this playbook right here you can see it's including my role that's what it does right here and then my role actually has nothing in it because it's empty so there's there's no tasks and then at the end it checks for item potent so it runs it again and as long as there's no changes so checks that there's changed equals zero and it didn't fail and if that completes successfully then idempotence test pass then there's a side-effect playbook that we don't care about here and then for verify it runs this verify playbook right here and then at the end it's going to clean up that environment and throw it away one thing that I do a lot when I'm building my in scible roles especially but also play books is I want to have an environment that I can dive into and look like when I have a configuration file that I'm editing I want to jump in there before that configuration file is edited to see what it was before and then I want to edit it and then I want to edit it with ansible and then I want to see what it is after and all that kind of stuff so if you run molecule tests it'll create the environment run the stuff and whether it passes or fails it tears it all down and it's gone and you're like okay I guess I can't do anything now because I don't know what failed so one of the patterns that I use a lot so I'm going to change this role in the role we're going to make my role install Apache it's going to say name install Apache and then we'll use yum since run sent to us named httpd state present alright so now we're gonna have install Apache installed on this server so I'm gonna say something different than molecule test I'm going to run molecule converge and that does everything that molecule test is doing but the the main benefit is it will stop at this point it won't run the item Potence test it won't destroy the environment it won't run the verify but it'll run my playbook it'll run my converge playbook and it'll run my role and I can stop it at that point and start looking at what I'm doing so if I run the molecule converge it'll run all the way up to that point so it'll first create the docker instance that it's gonna run and then after the instance creates which should be pretty quick since it's already downloaded the image it'll run the converge playbook and we should see that it installs Apache at this point and that might take a few seconds I know in my old laptop this would have crushed it and it would have completely died at this point all right okay so at this point it stops and if I look at docker let me make this bigger so you can see it if you look at docker PS you can see that there's an instance running here I'll make this wider so it shows up there's an instance here running and it just runs using while true to keep this container running so molecule can keep working on it I can run molecule converge again and it'll just run that PlayBook again on it but the cool thing is now I can say docker exact - ite - attached to it and then the instance ID which is instance or I can use the container ID 331 so I'm gonna say instance and then in bash that'll drop me into the container and now I can CD into my Apache configuration directory and I can look around it what's in there so CD you can fig D cat welcome dot config so if I wanted to I can grab this out of here I can put it into a template I can change things so I can dig into the server while I'm managing it and another thing that I do a lot while I'm doing that is if I have a full playbook that I know like let's say I do have a task that says copy source file that config best is let's see HTTP I think that B slash what is this welcome that config so I know that I have this coming up and I don't want it to hit there I'll just throw in a fail save that and then if I run this it'll get up to this fail and stop and then I can go into the container that I'm in look at the the files that are in there make changes change my playbook around and then when I'm ready see how it failed before it hit this task the number when I'm ready I just take this task out kind of like you know I might be crucified for being not not formal about this but it's kind of like throwing a breakpoint in your playbook to be able to debug things tossing the fail module in and using molecule converge so I often do that a number of times when I am when I'm doing this oh and somebody mentioned I did not know this let's see if this actually works you can use molecule login that is cool I had no idea I have a little show on my computer this is this is why I love the live stream so if I say I have a little Bosch alias set up for de enter that does the docker exec IT and then I can say D enter instance and I've been doing that forever and molecule login just does it automatically so let me exit out of here that's cool thank you very much who was that Mike Sawyer's thank you so you can use molecule login to drop into the instance that's like a million times better and that actually makes a lot of sense because molecule knows how to connect to it so molecule would be able to do that so you can log into it using molecule login keep using molecule converge over and over again drop break points using the fail module in your PlayBook where you want it to stop and and if you want also in the PlayBook you can use debug to debug what variables available so I could say debug see I can say register yum debug R equals yum so I can do that and throw it into a converged cycle to see what a variable has in it so it's it's really helpful for that it's it's like the the test the testing development feedback loop that I use locally and you can see that this actually failed because there's no file for that but anyway that so that's how I use molecule to do my local testing and that that test test develop and and fix things and go back into it loop when you're finished with it you can run molecule destroy and that'll delete everything and with all the other actions too you can run molecule dependency to reinstall dependencies from galaxy if you have any you can run molecule verify to run the verifier playbook all these different steps can be called individually to anything here so so next episode I'm going to talk a little bit more about how I use molecule with all my roles to run on multiple operating systems and we'll get into how to use molecule NCI and environment like it actions because I use Travis CI for a lot of things I use github actions for a lot of things and I use Jenkins for some things and I use Tower for a lot of running play books but I I'm not sure if if many people use tower with molecule to run tests and CI for play books but I typically use a CI platform like Travis or github actions but that will be for next episode and we'll also talk a little bit about an Sable's check mode and running against production and I won't have time in this livestream series for sure to talk about the full and the full option of parallel infrastructure because that's that's something that I could cover in like a two or three week long course probably but not in a one-hour livestream for sure but I am glad that you came today I hope you liked this episode and I hope that it has inspired you to maybe do a little bit more more ansible testing and get your playbooks to be more uniform more testable more more able to be shared with a team and and stable and maintainable and again I I mentioned that the maintainable playbooks presentation that I had I'll put a link to that in the description along with a link to the slides from the the other presentation I mentioned earlier and as I said at the beginning the video please consider if you can supporting me on github or patreon you can see my I'm I'm Gillan guy all over the place so just search for Galen guy and you can find me and that'll help me keep being able to do these videos and keep improving this book I put up a blog post last week that I for the months of March and April I I gave away somewhere around sixty thousand copies of ansible for DevOps and ansible for kubernetes and it's it's awesome and I also saw an increase in sales which is great but that's also sixty thousand people who might never pay for the book in the future so and I'm hoping that by doing some of this content and also you know being able to be sponsored on github and things like that that'll help offset any potential loss in revenue from that but thank you for watching next week it'll be the same time same place please subscribe to the YouTube channel and you'll be see when I start talking about this Turing PI cluster the first episode from that intro to clustering is already up in live so you can go back in my YouTube channel and find that the next episode I have almost ready to go I'm going to talk about setting up the hardware and comparing it to my my current existing cluster the Raspberry Pi drama which uses for raspberry PI's with power or Ethernet and then we'll be talking about running kubernetes with k3s on it so I am glad that you guys came today I hope that you are having a good week and please stay safe especially as things start opening please try to make sure that you're maintaining social distancing and and being kind to your neighbors thank you very much

Channel: Jeff Geerling

Views: 26,689

Rating: 4.9761906 out of 5

Keywords: ansible, ansible 101, devops, playbook, automation, molecule, testing

Id: FaXVZ60o8L8

Length: 64min 26sec (3866 seconds)

Published: Wed May 06 2020

Reddit Comments

[u/geerlingguy]

It's almost that time of the week again! This week I'll be introducing Ansible Galaxy before talking about testing Ansible playbooks and roles using yamllint, ansible-lint, and Molecule!

Paging /u/nixfu about the sticky situation :)

[u/tklisanic]

Do you plan to cover some testinfra stuff or will you use ansible as verifier?

[u/fr33d0ml0v3r]

I have been watching your videos. Thank you for the time and effort you are putting into them. I have found them to be invaluable.