Ansible 101 - Episode 13 - Ansible Collections and a Test Plugin

buddy - episode 13 of ansible 101 and my voice might be a little bit not wonderful today hopefully that's okay with you it's it's working enough that I think we can get through this hour I was very close to not having the livestream today because my health had somewhat of a dive last Wednesday Thursday Friday Saturday Sunday and Monday was finally kind of a turnaround day but I I was pretty close to going to the hospital over the weekend and was making plans for oh I could be there for a while because that's the way it is sometimes with Crohn's disease luckily that didn't happen I didn't test positive for the disease that shall not be named so that was another good thing and now I'm back with you today to talk about collections one let's say let me go over - where is it let me go over to the screen here and first of all I wanted to say thank you so much to everybody who's sponsoring me on github and patreon one of the things that has been so amazing to see for me is you know I have a job in a situation with open source work where I can kind of work at different times of the day which is great for me because I have Crohn's disease and sometimes it can cause you to have issues at different times a day and so you can't do the work that I can't work at normal hours sometimes and so I've never really been able to hold a job that would require me to be sitting in a desk from 9:00 to 5:00 or something like that so your support is is able to help me do these things so thank you very much - some of the new github sponsors like Chris Kahn me Zach biles Gary Mansell Barra hoskin not I'm terrible with names Parag Magoo Nia and John and and Maxim from on patreon and if you want to sponsor me there are links down in the description to do that my goal is to be able to get enough income from open source work that I can basically do a lot more of these videos and educational efforts and and be able to give give a lot more resources to people who want to learn about technologies like ansible kubernetes and all of the associated DevOps tools there are a couple things from last week's episode you might notice that I'm wearing a very bright red shirt today Solomon from I'm guessing who lives in Chicago said we don't like st. Louis where I'm from where I'm from so today I have my st. Louis Cardinals shirt on if you're if you don't follow baseball you might not know much about it but the Cardinals and the Cubs have always had a historically great rivalry but the thing is that the Cardinals are better so it's not really it doesn't matter if you don't like us we are the better team even if you have won the World Series more recently than us anyways handle barred also says systems that he used to work on decades ago and clustering was being rolled out there were failures in the cluster that caused more outages than if a single server with manual failover had been implemented and that was talking to my point last week about the fact that sometimes when you build systems to be more resilient and more more you know fault tolerant and high availability and all these all these amazing buzzwords that you might also sign up for extra support contracts and things to make happen sometimes that system is so complex that it's harder to run and results in more outages than if you had a simple system and so I usually try to make sure that I'm not doing something complicated in fact I'll give you a preview I have a new chapter for ansible for DevOps the book this is going to be the new chapter 7 which I hate doing this because now I have a new chapter 7 so I had to go back into all the chapters 7 and beyond and bump them and also change all the references in the book to other chapters and that means that some of the previous episodes where I talk about chapter 10 being about security well now it's chapter 11 so I did that though to maintain consistency in the book but at the end of this chapter I have a quote clarity is better than cleverness from Eric Raymond here I'll I'll hold that up and the camera of course there it goes the camera focused on it that's great anyway clarity is about governess I I like simplicity and my programming and in my code and infrastructure just because otherwise when things get too complex we're humans after all and humans can't we don't have the ability to go so deep into complexity even if we build AI and machine learning to try to understand that complexity we are human operators who have to deal with these things and if it gets too complicated it can be really hard anyways I liked that that mention and I think anybody that works with systems long enough knows you don't introduce complexity unless it's necessary at least you shouldn't someone Georgian also asked if I've considered using in next unit of computing and nuc or nook for kubernetes clusters instead of pies yes I build kubernetes clusters out of many different types of equipment and I'm gonna get to that in a future episode of my entering PI cluster series why would you build a cluster of pies when it's slower and in some ways less energy efficient per unit of computing in terms of CPU and memory and i/o speed and things why would you want to do it out of pies instead of something that's faster and more cool and hip and such I'll get to that but the bottom line is that it's more fun and interesting to do it with the resource constraints in my opinion than to do it with vanilla machine or VMs and things like that anyways although that goes a little bit counter to my previous statement that you want to reduce complexity using slower machines that are not necessarily as capable to do something might not be as as simple as using one computer to build a cluster or something like that and then Laurie asks will you speak about ansible container I will speak about it basically that is a dead project that I would not recommend touching there are other ways to build and maintain docker images using ansible or pod man images or whatever kind of container images you want the ansible container was an early attempt to see how could we manage containers with ansible and it was also somewhat I failed attempt it it didn't really have a lot of uptake and it didn't the maintenance kind of died out of year two ago at least might have been 2017 or something I don't know it's been a long time since it's really been been used anyways if you want to put where you're from in the chat it's always cool to see where everyone is around the world I noticed somebody mentioned the earlier that it's chilly in South Africa I would take that it's been hot and humid here in st. Louis feels like a swamp when you go outside there was one day last week that was actually pretty nice but I would I would like a couple chilly days where I could put on a jacket and thank you very much Hans for the for the little super chat donation thingy someday I might build some ansible automation to do something when there's a super chat in my office that would be kind of cool and yeah Yorkshire's too warm yeah Saudi Arabia might be a little bit hot right now too as well I've never been to the Middle East but I would I would like to visit there's plenty of places there that we read about in our kids geography class that now I am like Jordan in particular that I'd love to see it and explore anyways that is not what this episode is about this is about ansible collections and a couple episodes ago we talked about ansible galaxies and ansible roles in collections have some heritage from there and i-i've this this new chapter of the book you are the first people in the world to see this because it is not part of the book yet I'm still editing it and I'm still adding the last section talking work more about galaxy dependencies and collections and things like that so that I was hoping to have that done by this morning but like I said I was sick and when you're sick things are a little slower to progress like all of the different projects I'm working on right now they all kind of took a nosedive this past weekend anyway ansible roles can help you organize like ansible code ansible tasks and handlers and variables and things like that but they weren't ever very good at letting you organize anta modules and plugins basically the Python content that makes that you can use to extend man Sable's behaviors so a few years ago I'm kind of getting a little bit ahead of myself here because in the in the chapter that I wrote this is actually not the first part but let me let me get to that part just so that I have my my notes a few years back this was probably 2016 or 2017 2017 or so people people started saying like how do I share my custom modules with other playbooks and how do I share them if I want to contribute a module but I I don't necessarily want to go through the work of getting it in the ansible core which already in 2016 2017 we were seeing problems with the the ability to maintain all of intervals at that time hundreds I don't remember when it switched to thousands but that might have been 2017 the the number of modules in ansible core had just ballooned so some people were like I just want to have some modules that I put out there and maintain but I don't want to go through the process of getting them in core I don't want to maintain them in core it takes a long time to get my fixes merged into core so at that time we started exploring different ways of distributing modules and plugins and other things besides just rolls on ansible galaxy and the first attempt at that was building this tool called maser and maser this is something you won't really need to know if you're using collections but it's good to know the background Mazur was just an early attempt to see what can we do with what we have at that time which was a couple years ago against about 2.7 might have been the first time that Mazur started to do anything useful and then 2.8 was when Mazur could actually interact with ansible and with galaxies and put things onto galaxies that were not just rolls and Mazur lets you put what-what it was calling a collection of ansible content together and that collection could have roles and plugins and modules at that time and long term it was also envisioned that it would have playbook so you could have like a whole if you had a project France appeal you could put basically the whole into a collection and the collection could be distributed on the instable galaxy or at some point in the future in a private ansible galaxy that you maintain so maser was was an early attempt and it was just kind of experimental trying to figure out how is this all going to work how are we gonna put things together also how do we improve on galaxy's ability to have versioning semantic versioning and updates and upgrades and it had an idea of a lock file that lets you you could specify different version constraints and then the lock file would would give you a specific version when you install so some of these things actually made it into ansible core other parts of it are still not in the ansible core but maybe someday but if you want to look at the history of it this repository has some of that history in it and that I also thought it was interesting and cool because it was another Ender's Game reference mazer rackham was one of the characters who who kind of helped figure out how to defeat the buggers in the book so anyway if you've ever read the book he's he's an interesting character I guess at least but I thought it was also funny that in the in the description it said amazed there is also a hardwood drinking vessel it's like this throw that in there for four extra kicks I guess anyway so that that's maser it was the first attempt to figure out what to do and in the meantime it was obvious from anybody that looked at an Sable's community that things were problematic with the way that ansible is maintained ansible core itself in the way that modules and contributions to ansible war were handled because over over time as more and more companies started using ansible and more and more people started using ansible and it became literally one of the most popular open-source projects on the planet and definitely on github the backlog of issues started to go way way up and part of the problem was that now instead of just having a few hundred Linux based modules you had a few hundred Linux based modules a few a few hundred modules for certain applications a few I don't know how many thousand modules for networking appliances and devices hundreds of modules for Windows were added hundreds of modules for security things were added and so all of a sudden we had I don't know what it is what it was with 2.9 but like something like three or four thousand modules and the core team didn't expand as much as this this backlog starting the spanning so the problem was with any I mean we've seen it with with the pandemic if you're r0 is above 1e the backlog will just just keep increasing exponentially or at least increasing at some rate if it's below 1 then you will be able to whittle this down and get it back down to 0 but that has not happened for three three and a half years now so this blog post is by Greg Sutcliffe who's a data analyst for Fran scible and works at Red Hat and he has a lot of other good posts talking about how how we can measure sustainability in our growth and I've talked with him but in the past about this and he has a Twitter feed to add Greg I forget his name on Twitter I think it's linked somewhere on here anyways you can you can look him up Greg Sutcliffe but he has a lot of good work on how can we measure this and how are we going to measure this now that we have done some things to change this this track and make it better how are we gonna see if we're making it better or if it's actually getting worse over time and and so he's trying to set up data to help support that and show like maybe some of these new things that we've done we need to change course on them or or shift how we how we think about it and what was done is the ansible team looked at looked at the idea of collections which was something that was already kind of becoming a thing and then they also looked at how can we take ansible score development which has one cool feature of having everything under the Sun included in ansible core how can we make that more sustainable because having every single module in the world in one github repository with one team managing it even though that they had built some tools to make it so some modules could be approved by other people and and all that it just became super complicated and that's not really the way that the works with with github and with open source development cycles so the idea was we can break things out and take modules that are related like there's no reason to have a networking module for a Cisco device which requires some specialty to use and you probably aren't going to use it if you're let's say a PHP developer but a PHP developer wants a composer module to help them manage their PHP dependencies there's no reason to have both of those two modules in the same github repository using the same development workflow and the same team managing them so instead of that all of these different modules were were pulled out of the ansible core and this was actually done very recently I think a month or so ago was on maybe two months now was when all these modules were pulled out into separate content collections and so there's a networking collection and there's a general community collection and there's a kubernetes collection which I happen to maintain and there's there's all kinds of collections that have all the different kind of content and the idea is that over time we're going to try to get these collections of modules and plugins to be maintained by the groups of people that actually use them so instead of having one person who's not an expert in PHP and Cisco Networking switches that person if they are an expert in Linux Linux POSIX environment they could manage that POSIX collection instead of having to manage everything else so this is the big thing that's happening with ansible 2.10 the good thing on the backend for those of us if you've been following along this series you've noticed we haven't really talked about collections much if you're going to install ansible like we did earlier in the series using pip install ansible pip 3 install ansible you're gonna get the same thing it's it's not going to be different in the ansible 2.10 you're gonna still have all the same modules in that distribution but all those modules are going to be coming from all these collections and they're gonna be managed separately so that for example with the kubernetes collection since I have some expertise in kubernetes I've been able to merge some things like a new home module that we're sitting in the issue queue for ansible itself probably for a year now we put that into the kubernetes collection so now that's in there and you can start using it now that couldn't have happened if we continue down the path of having thousands of modules in ansible core so long-term that's probably going to cause a few speed bumps here and there and I would encourage you to test anything on the ansible 2.10 before you just blindly update to it and start running your playbooks with it but but it's it's gonna be a good thing for the overall ecosystem because we were not it was not sustainable to continue having the backlog increased by hundreds of issues a month indefinitely that that's not good for anybody to have that happen and it's a good problem to have that you have so much interest but it's also not going to help anyone long term to have that continue so it's it was necessary and it was it the hard thing was that because it it was becoming such a big issue to have all these modules in ansible core it was kind of done a little quicker than I think is ideal to get everything out and start moving to these new things so we're still I think as a community as the ansible community we're still dealing with how how this new world is working together it's a little bit like changing your tires what does it change tire well car driving it's a little bit like this this video here which I'll mute so that I don't get demonetised but you know while the cars driving we're changing the tire it's not ideal that's probably not the best way to change the car's tire but it can be done and it as this video highlights it's probably not the safest thing in the world but it can be done and it kind of has to be unless we wanted to just stop the car which we don't want to do we don't want to stop innovating and making automation better so anyway that is the history of ansible collections and to show how collections actually work I'm going to create a plugin which we have not done yet we've been using plugins and we've been using modules quite a bit in ansible but for illustration purposes I'm going to create a filter or not a test plug-in a filter plugin is also pretty easy but test plugins are one of the easiest plugins to create and manage and ansible and they don't require they don't technically require much documentation so it's it's it's easier to document and and demonstrate how they work so I have this playbook main panel right here and I'm going to let's say it's a front-end web development playbook that generates a web page and you have to choose a color for this web page and so when you choose a color you want to make sure that that color is a certain form of let's say blue in the sentences because blue is my favorite color so I'm going to save ours my color choice blue and that's what I'm going to choose you could have a red page or a pink page or orange page or something like that but I'm gonna make a task and since my my company's marketing team says no we really want to make sure that it's a form of blue and not red or pink or something else I'm gonna verify that my color choice is a form of blue and we want to assert that and we've used assert before just to make an assertion about our code it's kind of an in line test and ansible and what we could do here is say that we could say assert that my color choice is blue and that would be a very simple way to do this and I think that's gonna work let me remember that I have not tested as much of this because I've been sick so we'll see if we'll see if all these assumptions work let's see ansible playbook made that Amal that's this playbook and I'm going to use the the local connection because this isn't actually operating on anything it's just using an assertion so see local to use a local connection to my computer I'm going to give it a inventory of localhost this is kind of a hack in case you ever if you ever want to run a playbook against local host you can just throw it this and it will do that a little little hack for you to keep in mind I don't remember why the comments necessary and maybe it's not anymore let's see my color choice is blue oh this has to be a token so like that I'd leave name got string if blue wait a second maybe we do equals okay there we go so my color choice is blue the problem here is that this is a webdev thing my favorite color is not red Doug unfortunately red is good color although if you're if you're on video it's it's often the bad color for that because it over saturates the the red channel as you can see here it's kind of orangish over here reddish over here part of that it's the color temperature of the lights in my office but anyways blue is my favorite color and probably not this color blue though this is like that that garish true full blue I like more of a navy blue not quite navy blue but like between that and this anyways besides the point completely this is a little bit naive because what if we say what is what's another way to express blue we'll say Oh F F and that is another way to express blue so if we run this now it's going to say this it's going to say it failed but this is perfectly valid if you're talking about the web and you're trying to describe the color blue so we could say okay we could say that my color choice equals blue or my color choice equals this alright so that that would work but now we get a some some hip-hip developer who knows all about shortening these things up you don't have to have it if it's a color like this you can you can basically chop each one of these in half and you can say Oh F so that's actually another way to represent blue and it's gonna fail so yeah it can keep doing this and put it equals Oh F and do that the problem is whenever you start doing this I think I've mentioned this before in one of the live streams if I start seeing programming in my ansible playbooks that's a sign that it's a good time to start thinking about writing a custom module or a custom plugin or something like that to deal with this situation because we don't want our playbooks to turn into logic that's not a lot of people are like ansible is horrible it's programming in UML it's like well if you're doing that in my opinion you're doing something wrong and you should probably fix that by taking the programming out of the gamal and putting it into where it should be which is a programming programming language like Python so what we can do is we can write a test plugin for this so we can say instead of all this is say that my color choice is blue and then we can write a test plug-in that called blue and that blue test plug-in is going to take all of the different possibilities and and allow us to just write this instead which makes our playbook a lot simpler and when someone comes up and looks at this they're not gonna think oh man he's he's doing a lot of programming in this playbook they're gonna think oh this is really simple like they're just making sure that before we start building this webpage out of the color blue they're gonna make sure that it's actually blue so to make that plug in there's there's a lot of different ways that you can that you can write plug-ins and modules to hook into ansible and extend that's somebody says blue da ba dee da ba doo now now that song is gonna be stuck in my head the rest of the day from eiffel 65 there's a lot of different types of ways you can extend ansible as plugins and the simplest way in a playbook to add a plug-in is just make a directory for it so I'm going to make a directory called test plugins and it's the same thing for any other kind of plugin so there's there's filters like Jinja filters like you could say like this is a filter int you could make a ginger filter and those would go into a folder called filter plug-ins for modules I those are the one weird one where I think you those in the library folder and the documentation for all this is on interval documentation writing plugins something like that no and just developing plugins so you can find all the documentation about all this here we're going to make a test plugin so we're gonna do that here but up up towards the top it tells you where where to put plugins so that your PlayBook can see them and it gives you notes on all the different types of plugins test plugins are one of the simplest and that's why I'm choosing it here because we can actually fit it in fit it in an episode so I created this folder for it and I'm gonna create a new file called blue dot py and it's a Python file now you don't have to call it blue dot pie you can call it anything and instable will load it and it'll load all the tests inside of it I like to have my plugins all be in separate files though and named by what what's inside of them if you look at an Sable's core test plugins you'll notice that they're all kind of lumped together to go to an Cibola slash instable that's under what is it Lib ansible maybe plugins tests yeah so ansible has a few here and what an Sable's structure does is most of them so these are all different definitions for just functions that ansible a map to different plugins you can see here that it has a class test module this is the important part that returns a map of tests so ansible when it looks up what test modules are available to it it looks for any test module classes in any files in this folder and then it's going to call the tests the test function I call it a function but in python sometimes they're called what something else besides a function anyways it calls this method on the test module class and then you return a map of this is the the name of the plug-in and then this is what it's going to call it's going to call this function so in this case failed which is up here to find online 35 failed so it's gonna call this and then this is going to return whether it's true or false this particular test so we're gonna do the same thing in this this file and you can see that an Sable's code follows a certain syntax I think I believe that they use flake eight which is a Python standard to lint their code so this is all like kind of compatible with with that standard way of writing Python and there's also there's a few pepp standards that are used for writing in small code our little test plug-in is gonna be pretty darn simple so we don't need to worry about too much all of all of that linting and and how it's written and stuff but I do like to start my custom plugins just with one-line introduction of what is this doing so that when I come back to it later I can see like what is this supposed to be doing so I'm going to say instable custom blue test plug-in definition that's it's going to be in this file and then I'm going to do two different things one is I'm going to define have a have a method that is going to return whether or not a given string is blue so I'm gonna say is blue string this is going to be the method this is gonna be the method that is called whenever you call it the is blue plug-in this should also be for spaces so let me change my tab width here and then the next thing is as I said you need to have a test module or is it test module class right here with a tests a test method that returns the mapping so I'm just going to copy this out of ansible score and say this is going to be ansible blue test and the test module is going to return a list of tests return and it's going to just be one which is going to be blue so this right here this this this is gonna be the name of our plugin and it's going to be what you call when you do this so if I if I wanted to make another one for yellow I would just say yellow and then give it a function to look at or a method so anyway let's see I'm not gonna do a yellow one for now though so blue is gonna call is blue that's the function that's going to be called right here so at this point I could just say return true I believe is that let's save that and save this and I think this should work if not I probably did something wrong so let's see that worked but the problem is if I say red that's gonna work so we're making us yeah so that's going to work too which is obviously wrong so we need to add some logic here that says when I am given a string whenever this my color choice variable is passed to my function I need to I need to do a little bit of work to it to see if it's a form of blue or not so I'll write that really quick I'm gonna say blue values I'm gonna give a list of different valid values that blue could be one of them is blue that obviously should work another one is oh oh F F that should work Oh F that should work you can also represent it as RGB in CSS so I'm going to say 0 0 255 and you can also do RGB a but I'm not gonna put that one in because I'm lazy RGB you can also do % values so 0% 0% 100% and there's probably a library summer in the world that actually does more validations for this but I don't want to make this test plug in really really long and complicated because it's not even meant to illustrate all event Sable's Python capability that's more meant to demonstrate how to put a plug-in into a collection so I'll say if the string that I'm given is in blue values which is this this list right here then return true if it's not returned false and so now if I have read which I believe I still have yeah so my color choice is currently read this should return a fail which it does no test named blue what happened there there's blue is blue did I have a syntax error down here somewhere let's see it looks like it should be correct no test named blue why is it not picking this up in tests plugins Oh skipping oh there's a invalid syntax on line 13 what do they do Oh else I think I just need it cool in there yep so here's my my brain working in C PHP and Python together always ends up missing a little colon or something like that in my syntax and thanks Jose for noticing that so now we have a plugin that's working for this and if I change this back to blue it should work so let's try that and if I change it to one of these values like the RGB value for it that should also work and obviously this is not the most intelligent test in the world but it gets the point across that you can have some logic that tests something in your ansible playbook and and cleans up all that logic that could have been inside this playbook so we have a plug in here and let's say that we have this plug in and there's also a role that that we built that is going to template a web page and put this blue color in as the background for it or something like that it would be nice if we could package that together and then ship everything together and you could technically do that with roles a role on ansible galaxy can have a libraries folder and that libraries folder can have plugins and modules in it but there's a couple problems with that one is there's no way to list what's in that role in terms of plugins and modules so you can to know that things are in there and you can't search for it on galaxy and it can't be documented automatically that kind of this that kind of thing so instable collections offers a lot of those a lot of the functionality that you want let's see that and let me also make sure I'm not missing any any of my wonderful notes that I wrote to help me make sure I didn't miss anything like that : that I dismissed and let me skip over the history because I already talked about the history I might actually want to change the order in my book too we'll see alright so we want to put a we want to put together a new collection that this plugin can live inside and then we could also eventually if we wanted to distribute on an ansible galaxy and it also it looks like Hans is saying how many things are still Python - there's very little and ansible now that's still stuck in Python tool and everything that I've used in the past year has worked on Python 3 I don't have python 2 installed anywhere anymore so I don't think you'll have an issue there there there might be some old modules that still have issues but I haven't encountered them recently anyway let's see so what we want to do is create a collection for this and the collection can have plugins and it can have roles and it can have modules and we're just going to put this one plug into it but this will show you that kind of the structure of collections where they are how you call them from your PlayBook and all that and so what what we need to do is ansible automatically looks for a test plug in this directory for test plugins it looks for filter plug-ins for filter plugins looks for rolls inside of a rolls directory and it looks for collections inside a collections directory so I'm going to create a directory called collections and there it is and then ansible is going to look inside this folder for any local collections that we might have created and to create a collection it's the same thing as with the ansible galaxy to create a roll so if you if you say in galaxies roll in it and then give it a roll name it'll create a roll similarly we can create a collection with collection and let me move this up in case somebody pauses this and the playhead covers it collection in it and one thing that you have to keep in mind with collections is that they are a little bit different than rolls and that you have to have a namespace and you might wonder why it's because ansible uses a Python I think it's pep 420 or something it uses the Python namespace based auto loader to load collection code and because of that you have to have it in a certain folder using a certain pattern otherwise an Sable's playbook or an Sable's code can't pick up all the code in the collection so when you create this we have to use a namespace since this is a local collection I'm just going to call it local dot colors so this this collection has to do with colors I might have roles and modules that have to do with colors for my web app so I'm going to do this and then the other thing that you have to do and this is a little confusing especially when you're starting out with collections and it's again because of this namespace loader you have to create things in the right path for instable to see it otherwise ansible is gonna be a git I don't see your collection whereas with roles you can just name a role anything and throw it in the roles directory with collections you have to have it in the right path otherwise the collection loader won't see it so I'm going to give it let's see hey I'm on the wrong page in my book you have to give it a flag called an it path in it path like that and then I'm gonna say inside the collections folder that I just created slash instable collections so if I do that it's going to create a collection for me in here and this is you know this might if you're familiar with like Java or some even PHP a lot of projects are doing it this way now there's a lot of empty directories here and that's just necessary for an Sable's new collection loader to load all the code correctly so you kind of just have to live with it but you the collections all live in here and then they have to be inside ansible collection so that ansible can have it's like global namespace to load these things and then you have your namespace for the collection which is local and then finally have the collection which is colors and an inside colors this is actually the collection itself it it comes out of the box with a few folders Docs plug-ins and roles and then a galaxy email file the most important thing in a collection and really the only thing that's required you could have an empty collection if you want is a galaxy file and the first few things are necessary the rest of this stuff is optional but strongly recommended none of these things really have to be filled out if you don't want to for a local collection they all they're only really important if you need to put a collection on the in symbol galaxy so you can leave the version the same forever if you want if this is just local and but you do have to have a version in your galaxy ml for our collection so I'm just going to remove all this stuff that I don't need for this local collection right there and delete all that so this is all that you really need for a collection to be usable locally and then right now the the plugins directory Docs these are all empty and I I don't have any roles yet so I'm just going to delete that folder and I don't have any Docs it's probably good idea your rights and Docs but I don't have any for this particular example and then plugins has a readme that says set us how to use this directory so instead of having a folder called test underscore plugins in my playbook in the plugins directory I just create a folder for the type of plugins so I have a test plug-in so I'll create a test folder so I'm gonna do that new folder test so now I have a test folder in there and I want to move this blue dot PI into that test folder I wish you could drag and drop inside of sublime but I can't so what I'm going to do is I'm going to reveal this folder in the finder and then I'm just going to drag it and use spring loaded folders or whatever this feature is called to pop it in their plugins and drop it into test all right so it's in there now and it's not in it's not in the test plugins directory so I'm going to delete that folder now if I run this playbook there's going to be a problem I'm gonna whoops that I already did that let me clear this I'm gonna run the PlayBook again and it's going to tell me that it can't find the blue test anymore and that's because if there is a there is a plug-in down in here called Blue Dot pie and it should be picked up in this collection and ansible will automatically scan for collections in the collections folder but you can't refer to it the same way as you did when it was kind of what I call a bear plugin not not like a bear that roars and goes roar but a bear like be a re a plugin that's just in your playbook directly with collections you have to refer to plugins let me make sure I get the terminology right because I always say the wrong thing with collections you have to use the fully qualified collection name or you'll see it in the documentation sometimes referred to as FQ CN and that is basically the the namespace which is the namespace which is local and then the the collection name which is colors and then the actual plug-in itself which is blue and those are all separated by a dot so if I say local that colors that blue and I rerun this I believe that that should work unless again okay so it did it picks that up and again the reason for that is ansible for collections is using pythons built-in namespace autoloader I forget with that it's like pep 420 or pepped for something anyway if you want to read all about that you can you don't have to know how the autoloader works but you do have to know that for a collection to work it has to be in a folder structure that that starts with ansible underscore collections then has your namespace folder then has your collection folder and then has all the different plugins and roles and modules and things so that's when you're starting out with collections that's that I think that's the thing that's been tripping me up the most especially since I'm used to the félicité with a roll where you know you have a roll and you can name it anything you want you don't have to worry about namespaces and you don't have to worry about fqc ends and things like that with collections you always have to no matter what so that's something to keep in mind if you're gonna build a collection and if you're gonna put it on ansible galaxy there's some other things to keep in mind too not only should you fill out this whole galaxy AML file you can't just have a github repository and throw it up on ansible galaxy you have to build your collection and there's a tool ansible ansible galaxy collection build I think it is that can build a version of the collection as a tarball and then you upload that tarball using ansible galaxy collection push or I forget published I think it is and that publishes onto ansible galaxy or if you have a private ansible galaxy which i think is some feature that might be coming soon then you could push it there too and you can you can actually have multiple ansible galaxies that you can pull things from so those are all things to keep in mind and I was going to in this episode try to talk through all of that process but instead I recommend that the hard thing is that because a lot of this is new ish so again scible 2.8 was the first time the ansible collections themselves were a thing in ansible two-point-nine collections were kind of refined to the point where they were more useful and could could be worked with yes well 2.10 is not actually released yet but it has a ton of little improvements to the collection user experience like you know if i wanted to show you some of the things i want to show you I would have to caveat a lot of things and say like this is how it works now but an answer well 2.10 it'll change I don't want to have to do that in this video because I'm trying to make this this series be a little more a little more timeless and apply for a little longer period of time but I will talk about a couple things in particular if you go to ansible galaxy there are already some collections on there there's most of the collections that are parts of ansible core let me sort this by download count just to show some of those so so the kubernetes collection has the kubernetes modules that were taken out of ansible core 2.9 and put into this collection and then they'll be in the ansible distribution in 2.10 but they're not part of an Sable's devel branch they're not on github any more in ansible they're separately maintained in the kubernetes collection similarly there's net common has common networking modules that are used by all the different networking systems there's a cloud it's Google cloud it's it's kind of funny because it's like oh cloud and then you're like oh it's Google cloud that you know it so you might see like five cloud collections but one of them is Google's one of them's Amazon's so collections on galaxy the the usability of discovering them and also vetting which ones are the appropriate one for you to use some of that's also being worked out right now and that's that's another thing that's another reason why this collections episode has is more towards the end of this series and not at the beginning at the beginning this would have been even harder to do because some of the little things that I've been doing would have worked differently two or three months ago or I wouldn't have been able to explain it that well because some of the things that I would be explaining were things that I still didn't understand myself but let's say you wanted to use the kubernetes collection you can you can do it the same way let me see here I'm gonna create a new file a requirements tunnel file save this as requirements that Hamel and in here I'm gonna say collections and this is all based on memory I haven't finished this part of the book yet so I think it's like named community debt community that kubernetes and version will do zero point eleven point zero and if I say ansible galaxy install what this is going to do is this will install this collection by default it won't install it playbook local even though it'll pick up collections playbook local like this it will install it by default into not a global path but a path that's that could be shared with other playbooks I don't really like sharing collections between play books because especially with kubernetes one playbook I might have written with kubernetes 0.10.0 and I don't know if it's going to work with eleven-point oh yet so I'm going to I want to make sure that those play books are different per or these collections are the version that I specified per playbook and and because of that I want to install it locally so would I do a lot of times as I say you add an instable but in ansible that CFG file - can config telling it suppose some configuration defaults to override so I'm gonna say defaults collections paths and then I'm gonna put that equal to collections and what that's going to do is tell ansible galaxy when you install collections install them into this folder so instead of installing this kubernetes collection into my more global user path or into user local or user share wherever it installs more globally it's gonna install it local to this collection or local to my ansible playbook so I'm gonna say in scible galaxy install - our requirements and oh this is another thing about collections so I I think this might be fixed in ansible 2.10 notice that I I just did this by muscle memory I wanted to install all the requirements so I ran this command and nothing happened that's because ansible galaxy install currently requires you in ansible 2.9 to do roll install and it defaults to roll install but it didn't see any rolls in here so there's just collection there's no rolls so it kind of just did nothing and that's it I opened an issue for this saying this is basically a bug because I had no idea that nothing happened it's up and there's no if I say I echo the exit it you know it was successful so I would imagine that and install that but it didn't in ansible 2.9 you have to say ansible galaxy collection install and that is today I haven't hit enter there we go and that is actually going to install the kubernetes collection and so if I look in here now there's a new namespace in my collections folder community kubernetes and so I could start using that kubernetes collection so as I mentioned this is why this is way later in in the series I wanted to make sure I had a full understanding of how this works before I fumbled over at all live on a live stream and in ansible 2.10 some of these things like like this I believe works in two point tenets it will basically install all the roles in all the collections without you having to run one command and then run another command later and like I said I like to have my collections in the playbook installed in the playbook and then you can also make the decision if you want like let's say you're using ansible tower you can say I want the collections to be in the source with my repository and I'll manage them on my machine I don't want tower to reinstall them every time because collections some collections have a lot of dependencies and and what happens is if you install like community general and some of the other big collections it can take a minute or two to install all the collections that you might depend on for a big playbook so because of that what I usually do is I commit all of the dependencies to the repository and then I just have automation that updates them whenever I change my requirements file and an updated version or something like that let's see is there anything else I wanted to cover really quick here another cool thing about collections versus roles is on the ansible galaxy what happens is you build an artifact of a collection and galaxy hosts that artifact and that artifact that that tarball for each version doesn't ever change and with ansible rolls what I could have done if I was malicious and and didn't like you and you used one of my roles is I could have changed the contents of one of the older versions of my role that you might be using and it would change like you would install it and it would work great but then I could change it later because it was pulling it from github and on github you can change tags and what's inside of a tag so I could have done that and that could have caused problems for you with this ansible is actually hosting the the collection that you're downloading and all the content that's inside of it and that that also helps with the potential for something I think it was left pad in nodejs or with that chef issue if somebody if I were to delete my collection off of github entirely you could still install it because the artifact the the thing that was was released as version 0.9.0 that's hosted on ansible galaxy so that's a cool feature of ansible galaxy and it's also all powered through a back-end called pulp and Pope is like a Python dependency any managing packaging system thing and because of that there are some other features that are a lot easier to have a path to implementation like upgrading your dependencies and uninstalling and reinstalling and doing things like that that we couldn't really do as much with roles on ansible galaxy so there's there's a lot of cool potential future growth and there's a lot of things if you if you go to the in Sable's repository or galaxies repository and you filter by author gear leg guy you'll notice that all these little things where I'm like oh we could do this oh we could do that there's a lot of those kind of things out there and I'm I'm excited about it I think it's going to take a little time because the the move of all of ansible x' modules and content basically taking all the batteries that are included inside of ansible pulling them out of ansible and putting them elsewhere and then building a new system to pull the batteries back in for when you install ansible that's been a lot of work and that's you know it's hard to polish things when you're also trying to make a lot of foundational changes at the same time if you've ever done any building projects you realize when you need to work on on the foundation of a building usually you create a very huge mess and you have to rip off a lot of siding and a lot of the nice stuff that looks nice and how and then afterwards after all of its done and after the foundation is nice and strong and set again then you can start working on patching up everything and making it look nice again I think we're at that stage at this point with collections and again another reason why I didn't want to try to cover collections earlier in this series and I probably will talk more about collections in the future even outside of the ansible 101 series just because there's there's a lot more to them and they can help somebody mentioned from a security perspective to there there's a lot of security benefits the way that collections work and the ability to also download them like this offline and put them on to servers without an internet connection that kind of thing those things are a lot easier to do with the collections and they were with rolls anyway let's see collections the new NPM yeah the hardest thing with with all these things its dependency management is so hard it's hard to do it right and then it's way harder to do it right and make it usable and there's always a lot of trade-offs to make you know performance versus security versus usability and I've never met a dependency manager for any language that I'm like man I just love this there's nothing bad about it there's always gonna be something difficult or something that's not implemented as well as it could be or something as slower than you want I think with collections the thing that the thing that bugs me the most is just the fact that you have all these extra empty directories and it's like if I want to get to if I want to get to actual code it's in alignment collections I have to open collections ansible collections to empty folders so far 1/3 empty folder which is the namespace and then finally I had the collection which is here and then inside of it I can get to the plugins and things so I get why it's done it's it's just it's a little I don't know it's it's slightly abrasive to me because that feels more like programming where you have namespaces and things like that that are required for everything you know if it's local so what I've done a lot of times is for my play books I still have roles that are local to a play book and a roles directory and plugins that are in the test plugins or in filter plugins I still have them local to the play book if it's just for one play book I don't usually use collections for that just just because I don't want to have to deal with the folder hierarchy you could do it either way the other thing is that in the play book itself you know you don't have to use the F qcn if it's a local play book plug-in I can just call it blue and that that's it's a little more semantic for me another quick note on that before we go you can it doesn't work for test plugins but if it were roles so let's say we had a role in that collection so he said roles local that colors that make me color or something like that and that was the role name you can add a keyword up here at collections and say local that colors and then in your play book you don't have to refer to the collection the fully qualified collection name it ansible automatically a Lea the it'll it'll look for this role inside this collection but that carries with it its own caveat and its own set of potential downsides especially if you have two two collections that define the same role which can happen like an installer role so that there's there's a lot to there's a lot that still needs to be hashed out a little bit and the best practices that will be made and hopefully everybody watching this can can help make some of those best practices happen too let's see any good questions really quick before I wrap it up commit hooks something someone didn't someone actually post it in comments on YouTube about a commit Huck for linting ansible I saw some stuff on Twitter too you can look that up on on Google but you can have a commit Huck that will lint your ansible playbooks binging playbook done with molecule cool very cool first time on live episodes welcome very we're glad to have you my my tire Matty test verifier so I haven't talked about ansible tests ansible test is a tool and that is instable tests like that or I could just do it here the ansible test is a tool for testing ansible collections and ansible content and it's used by ansible core as well it's similar to molecule in some ways but it's also a little more it's a little more for the Python side of testing but I would look that up on online I actually have a blog post about testing collections with ansible tests you can test collections with molecule as well and I have a blog post on that on Jeff Garlin comm anyways if I didn't get to your question I always glance back at live chat so I'll try to get to it next episode you can follow me and consider supporting me on github or patreon if any of this stuff helps you thank you very much for that you can follow me on Twitter which is over there I'm girly guy most places in scible galaxies pretty much everywhere and thank you so much for watching a-and I'm glad I was able to do this I I was very close to canceling and that would have been the first week in three months that I wouldn't have done a live stream I don't know what I would have done with myself anyway thanks for watching I will see you next week where next week I'm going to see if this is gonna work it could fail spectacularly so please tune in to watch it'll be exciting one way or the other I'm gonna see if I can talk about ansible on Windows and I'm gonna use a Windows laptop to do it I'm not gonna livestream from it I think I'm gonna try to get the Windows laptop into my Mac and then stream from the Mac while I'm doing the work on Windows so that because the Windows laptops a little bit older but we're gonna talk about that and see how far we can go talk about Windows subsystem for Linux talk about the Windows modules for ansible all those different things so please tune back in next week and thank you for watching these ansible 101 series I hope that you're having a great day you

Channel: Jeff Geerling

Views: 12,010

Rating: 4.9711189 out of 5

Keywords: ansible, ansible 101, devops, playbook, automation, collections, galaxy, plugins

Id: nyXDR4RG4A8

Length: 62min 20sec (3740 seconds)

Published: Wed Jun 17 2020

Reddit Comments

[u/geerlingguy]

Pinging /u/nixfu - I completely forgot about this week's episode!

[u/scritty]

I'll need to modernize our roles and custom modules, looking forward to watching this one.

[u/wutanc]

Just found this, not sure if I’m upset or happy atm, having to watch 13 episodes. Bye bye weekend.

Well done tho! Looked at the first 15 minutes Of this episode and it seemed really well done.