"One person's Ansible is another's pyinfra." - Daryl Tester (PyCon AU 2023)

foreign 's career has spanned two centuries if not millennia he has worked a variety of roles within the I.T industry and one day he may even pick one and stick with it today however is not that day Dave is going to tell us about how one person's ansible is another person's pie infra take it away Dave Daryl sorry [Applause] um I'd like to pay my respects to the traditional owners of the land on which we meet today the Ghana people of the Adelaide region and pay my respects to Elders past present and emerging I'd also like to thank our interstate and overseas visitors for visiting my home state it's lovely to have you here and you should definitely come back when it's a lot warmer preferably when we're not on fire though as an ex CFS member I'm qualified to say that I would also like to acknowledge my employers if they're watching um technically I'm self-employed but the people pay me to do money people pay me money so to do things to their computers and My Views probably don't represent theirs um and I'd also uh would like to ask for questions at the end of the talk um I'm a little bit nervous up here this first time for me so quite easily derailed in my thoughts I'd also like to add to that the GitHub repo isn't ready yet um the talk will be uploaded probably in the next couple of days so uh don't hit it yet so I'm going to make some assumptions um that you probably all got a passing understanding of python and you probably all got a passing understanding of interval which was why you're here if you're thinking this is an ansible beginner's talk not quite um and I'm not really seeking to convert you I'm just providing an alternate point of view so previous talk titles that I've wrestled with um is uh yes what you see here these are the the more family friendly ones um so and to be clear these are talk titles that were rejected by myself not by the program chair the program chair here has been lovely um I've been struggling with the theme of this talk and it's something I wrestle with constantly um because I'm very uh OCD about diving into problems and problems that I perceive so this is kind of what that talks about but I still haven't got basically the essence or the Zen of this talk out yet so um still a work in progress so it's also sort of documenting my struggles with ansible over the years and so which will lead me I guess to confession time which is what was taught currently kind of is um I struggle with ansible like I've been using it for probably five years now and not what it does or what it is but more how it does what it does I find ansible tends to be a tool there are some tools that sort of fit into this model where you're trying to solve a problem and you have to think about that problem in terms of the tool that you're using against it and and I find that quite often that was my case with ansible is like I'd have to structure it in terms of the way ansible thinks too to do things so and it sort of it it caused a fair bit of cognitive workload on myself doing that just having to deal with then I won't call them shortcomings they're just it's the way ansible's been developed and grown over the years um and it sort of it takes away the workload from actually thinking about the problem itself some of the uh the cognitive workloads that I mentioned um keeping track of global state that sort of makes me feel like my old GW basic days and to be fair I came to GW basic after I'd worked in other Basics that had things like local variables and name subroutines so it was going back and trying to have to think of global variables again uh and winter and winter not Ginger Ginger coding still kicks my butt on a regular basis uh the obtuse error messages and this is a true saver message is more a case of um if you've forgotten to download a collection that you're using you'll come up with a rather bizarre error it's sort of it's a the equivalent of a code smell but in the design space because it's a parsing issue when that happens and it sort of it it again when it occurs I have to think about it and think why are you doing that is that something I've coded wrong and it's like oh no I've just forgotten to incorporate the correct collection um data structures I tend to a lot of my playbooks tend to be well complicated playbooks tend to be build data structures around how I want things to be deployed um if when they start getting moderately complex and coupled with uh templating it gets pretty messy pretty quickly include task versus import tasks I struggle with that all the time I can never remember when to do which I've always got to look up the documentation for that nested looping as probably also my favorite is there is many ways to do it in uh ansible not necessarily every way is equal there are just some dead ends that you'll go to well they're not that ends until you've actually gone down them and then you'll realize no I should have handled this differently so so I I tend to think long and hard about these things I spend a lot of time lying awake at night staring at my ceiling um I shouldn't do that I should have hobbies but I'd only waste them uh so while I was sitting there and again I tend to like research these things and trying to dig into the nature of what it is um somewhere between giving the previous version of this talk and this one which was probably about a year ago now in the python local Adelaide users group I came across this quote um which sort of sealed the deal for me it was like yeah it was it was quite an opener and it sort of it did actually appeal to me um it was it was pretty much a light bulb moment when I came across this ironically the coincidentally the uh question talking about this was talking about ansible uh and I wasn't actually researching ansible stuff in the time I was just having to be looking for something else but when I came across this it was just like oh okay he's studying he's suffering from similar issues with ansible and I was finding there's actually a small group of us like that that are struggling with some of these issues this is sounding like a self-help group sorry about that um and coincidentally uh he while I was sort of looking for a new product to pick up I'd already picked it up by the stage and he was also talking about the same product hello certainly so you have your my permission to touch me okay it's me it's my uh well now I can hear myself that's worse please don't do that I'm sorry that's okay it could be my metal implant maybe I'll just try not to move so much was that better I would just this is going to be awkward um so I was reading um lox blog post um he was talking about problems with ansible but he was also saying oh I've also found the solution which I'd already started dabbling in and by dabbling I mean actually wholeheartedly committing and the thing in question was Pi infra when I came across Pine for I sort of went oh because it's rare that I do that for a project these days um but it was hitting all it was solving a lot of no solving's probably not the correct word but it was covering a lot of the pain points that I was having with ansible um so uh basically it's very similar it runs in the sort of the same problem space as ansible it's uh an agentless orchestration tool a impotent for the most part um you can still sort of work around that but that's something you shouldn't do and it has inventories and playbooks well the equivalent of playbooks um but they're all written in Python it's so generally it's it's it's a similar model to python it runs basically on a control node sorry python ansible however it sort of differs from ansible ansible tends to upload well what tends to ansible compiles a module uploads it to the remote end and runs it expects you to have python on there Pi in for a dozen buying for just expects a working shell of some sort at SSH is across and not necessarily just SSH it has a whole bunch of connectors but SSH is one of them and is the default one executes a bunch of commands on there copies the results back or retrieves the results back and then processes that and then continues on from there it has facts for doing a lot of that so it will use the fax to determine the current state if you structure you deploy correctly it'll use fax to determine the current state of the system then apply a set of operations to bring it to the desired state that you want so this is where the eye dependency comes in high dependency sorry practice that word and I still have trouble speaking it out loud so this is where the idepatency comes in where you would sit there if you re-execute a Playbook and there's no changes to be made it should not make any changes to that to the desired state of the system uh I was saying before it has connectors what it what it calls connectors so SSH is usually the bog standard one you use unless you're talking to your local machine and it's got a specific one called local for that basically a connector is any means that it can use to talk to the remote endpoint other connectors that it includes a connectors for Docker terraform and there's a few others as well which completely escaped my mind the SSH sessions remain open for the duration of the run if you run ansible and your playbook has multiple plays in it it tends to do an SSH connection for each one if you've got uh like some environments I've worked in previously um sorry waving to a colleague there X Colleen um uh for SSH connections if you're spawning off a bunch of them a connection create quite a sort of uh quite a workload behind um sorry that's me making noises again sorry I don't think I can keep still um if you spawn a bunch a large bunch of SSH connections you can actually create uh quite the workload behind on your authentication back-end system so uh with this it spawns the SSH connection but it remains for the duration of the entire run it doesn't uh sit there and reconnect it also opens ssh in a slightly different mode to what ansible does ansible is quite noisy if you do uh for example it logs in using a regular user login account so if you do a who on the system afterwards you can see you've just done a manageable run because your ansible user is logged in quite often this uses ssh in a uh in more of the remote connect mode like when you're typing SSH hostname command so it doesn't actually appear in your login but it still appears in your ortholog so if you need to go back and do any sort of research for that um so it's saying it uses the remote Shell at the far end it's not particularly wedded to the remote shell that you use it's got a variety of um different sort of plugins where you can sit there and and depend on your back end shell that you connect to you could be connecting to some proprietary device like a switch for example and uh and it's particular set of commands to use for that the deploy process runs as two passes so the the initial run um sits there executes all the code gathers all the facts that it requires to do that and it builds up a set of operations and then the second part just blats those operations on top of the the system to bring it into the state that's required so I present you a concrete example um this is a lab maintenance script I call it a home lab maintenance script that I used so I run a small set of vps's and whatnot um I quite often SSH into them as we all probably do um but what I don't want to see when I SSH into them is that or usually I don't want to see the initial connect message that the initial connect message is usually you've never logged into this host before do you want to and it's like I think it's the host I'm connecting to I always have that moment of Doubt I've dagnabbit now I can hear it sorry guys um I always have that moment of Doubt logging into it and I know of a previous Linux conference where someone did do that and ignored the warning and were subsequently slightly pwned so uh for SSH key hygiene so I tend to now be quite security conscious I tend to make sure that I populate my global known hosts SSH keys with all the keys of the machines I'm connecting to so that when I do connect to them if I ever get a scary message like before I know something is really wrong and I never get the you're connecting to this machine for the first time sorry just what's up so host names this is uh good Lord my slide is incredibly small sorry I've got to appear around at this one um so this is what we're manipulating uh the top one is the name of is the the public uh known hosts also the public host key that the server will be known by and what we've got to do is get the contents of that into the SSH known host file the global one um slight wrinkle these days is that um we hash the host names because security I'm 100 sure why but it's probably if they do it it must be a good reason so we don't tend to keep the host names in the clear these days anymore uh the other key point there is that there's also the uh so the in the SSH known host file there's the uh the hashed hostname there's the key type and the key value and usually there's a bunch of key types per host so this is the ansible Playbook that I originally had for it um and to fit it I've stripped out no I've stripped out a lot of the comments and now when I go back to look at it I quite often find it quite difficult to read without the comments which is the part of the whole beef of this talk um so this Playbook basically it logs into the room well it uses uh it fetches from the remote host uh the list of files that we need to get actually prior to that sorry um we know the remote host by one of three ways we know by its fully qualified domain name it's fully qualified domain name with a DOT on the end of it and there's good DNS reasons for that which I won't go into here and then it's also known by its IP address which is the long convoluted host vars underneath it we then connect to the server and we get a list of the public key files that we want to bring back across and with a new slip to fetch them slip brings them back as a base64 encoded file because it's not uh it's only ASCII clean the connection so um the we retrieve the the key back actually we've achieve all parts of the public file The Next Step then splits it up got it's boring me even it's not boring it's a lot of detail and what I find and this is partly what I'm trying to convey is what I find is that with the comment strip I get confused as to what it's doing easily and I have to stare at it for quite a while and again it's the problem I'm trying to solve not the problem I'm trying to solve in the tool that I'm trying to solve it with um there's all sorts of chicanery in here there's the uh the use of default for the SSH host key because SSH host key when you do the original pass through it is blank or it's actually doesn't exist so if you don't have the default ansible then throws an error and saying this key has never been used before and it's like well I'm just looping through it typically if I was doing this in Python I'd just set the variable beforehand so I know it's default State and then proceed through it uh this goes on this then gets to the the the uh the uh the end of the wedge of the matter I guess um this then actually updates the known host file we do this on the local machine we have delegate 2 on localhost which says for all the data you've just collected from all the remote machines we're now just applying it to in this case it's usually the laptop that I'm running it from um we're doing a lot of looping we're doing looping with products and again if I don't go back and review the ansible documentation for I think what is it I'm doing here it just slips out we also have in Loop control where we can sit there and as it's as edible is looping through it it sits there and reports only on the subset of the information otherwise it just tends to blurt out great deal of noise onto the screen and if something breaks you're not actually sure what's broken so I come to there's there's some ways you can work around this like you can deploy you can modularize stuff to tasks but again once you do so you've got to sit there and you've got to be aware of what variables are going into and out of tasks you've got to track that yourself you know we have methods in Python where you can pass in local variables these are things I don't have to worry about these are just things that occur naturally to me when I'm coding it's like but when I'm coding an ansible I've got to like no I can't do it that way I have to track all of this state myself that's the computer's job I don't want to be doing the computer's job I've got other things to do stare at my ceiling so again as I'm obsessing over this um I love this quote um it's from an old web comment which no longer exists unfortunately but I've still talked to Phil Barnes occasionally Owens the program manager and Desmond's the programmer and this is one of the things I struggled with is like this is part of my struggle with this whole concept is I'm still writing code you know and part of ansible's reason was to be able to be usable by the every man sorry every person sorry that's gender inappropriate um usable by everyone it should be relatively simple to put together and for simple cases it is ansible was quite simple but when you get to start doing complex things with it it just for me and it might not be everyone here but for me personally it goes off the routes fairly quickly when I have to burn brain CPU Cycles on details that I'd rather be burning elsewhere so I was concerned when moving to python when moving to Pi infra but not just paying for it I was looking at other programs in the same space I was concerned am I just swapping one program from another one problem for another but I don't feel that's quite the case um again it's more the power of abstraction that you get from Python and being able to express problems in Python so I've converted that script I'm sorry that Playbook um Pi infra and as I said point for basically the inventory and the deploy scripts what its versions of playbooks are um for all intents and purposes python so first up is the inventory um for so for my example um we just this is the inventory.pi that we pass into paying for it's just basically a list of servers normally it's just a straight list of servers I actually like again I like my running dialogue as things are running to be reasonably cleaned so you don't have to pick through and try and pick through the error messages so I like my server name short and my SSH known assh hostname's long so basically when playing for runs it knows that the host that it has to connect to known as server one is actually server1.example.com and so on you can pass other uh variables in sorry you can assign other variables into this these are just the ones that I needed for the purpose the upshot this though is that it's executable python code so you can use something like requests and grab your inventory from you know a convenient Json server that's located nearby but has all your stuff there if it doesn't come back in the format you know that's that's like this you can Mudge it within the script you don't have to write a uh a separate inventory source as you would do with ansible you can actually do it it's it's python you can do it as is uh so paying for when running this it's just concerned about prod servers it can it basically any uh variable that's sorry any yeah variable that's set in the module space it uses uh as the inventory so you can have prod servers you could have Dev servers and they'd be selectable from the command line as well or it'll just execute against the lot uh uh I'll come back to it local shortly uh but at local as I said before it's uh basically saying I want to connect to the local server not virus sh but I'm sorry I won't come back to it just I'll come back to it right now uh the reason we do that is that Pi infra doesn't have the concept of a delegate to so when previous Playbook had a delegate to to say I need you to run this on the Local Host time for it doesn't have that but it just lets you run your playbook against localhost anyway uh and we have group files as well same as ansible so almost feature complete for that and this is basically saying this is the SSH user which isn't my user in case any of you thinking about trying to hack a new my framework um and the key which one of you have either but um be sure to commit that to uh GitHub later uh interesting bug and it's probably worth keeping and this will be uploaded to the repository um interesting but paramico and modern versions of Debian and Ubuntu SSH servers can't negotiate can't play nicely there's a couple of algorithms there that it seems to it tries to use and then dies badly but it's a simple matter to disable those and say just exclude these algorithms in your initial setup and negotiation and then go nuts um the last line uh that's also a paramico thing uh paramico just uses your local your personal home directory SSH known hosts by default holy duly I am so sorry have you wiped that five minute thing in me yet you did sorry about that um speed run I apologize for this because this is what I swear I wouldn't do ah the deploy script proper ly to the deploy script uh the deploy script only needs to execute on the local host and this is as you can see pure python um I grabbed some bits from paying for it that I need I also grabbed some bits from help script that I need and then I sit there and I go right I need to slurp up all the keys from all the hosts that aren't me hence the I name does not equal that local and the ones I've successfully connected to so if there's any servers offline it skips over them as well it grabs the connection IP the SSH hostname and then executes that fact SSH known host keys it then sits there and creates all the host names based on that so that's the fully qualified domain name with them without the dot and the connection IP and then bunks that all into a list of lists and then it calls this is H known host subset which sits there and says passes that all off to the SSH host module um which is the operation that Pi infra runs so it does that tells you the name of the path the path to the message you want to update name is the descriptive field that comes up as it's running so you can so you know what it's doing in the play and of course we need pseudo access to do that so just say give me sudo or give me death I have cheated slightly but to be fair answer we'll start at it ansible has a lot of building facts and modules that we can leverage on um that we do not have so I've actually had to write two facts and an operation I'm going to really speak on through this this is a fact these are very simple these are so simple I've spent no time thinking about writing these as it should be again this is the problem that I was trying to solve I do not want to be thinking about how to solve these problems I want to be thinking about the problem that I'm trying to solve in ansible sorry the automation problem not in ansible no um so a fact is basically we derive it from what's called fact base which is just the you know the the superclass there's two methods that we need to implement command which in this case we're actually there is a helper script that applying for supplies that just lets us go hey we just want to cap the output of all those files and then there's a process which it runs which basically grabs the output of all those files which is executed across SSH brought brought back and then we just go hey do with it whatever and what I'm doing with it here is basically just returning a dictionary that says this is the key type this is the key value imagine says one minute um but this is the thing is that the fact is structured in a way that helps me solve the problem that I'm trying to solve and skip over the next fact should note right and then this is the operation the operation is basically it grabs all the data that you've collected and then sits there and where what I'm doing here is irrelevant skip that um but we need to basically say in the SSH known case files we need to delete Keys we need to append Keys we sit there and we work over we do basically technical Mumbo jumper don't need to care about it but what we do at the end is we go for the lines that I want to lead out of the key file this is the said command I want to execute for all the keys I need to append these are the keys I need to append and if I've made any of those changes it's almost up if I if I have made any of those changes then I need to use SSH Keygen to sit there and reformat the output to rehash the output and I think I'm done and I apologize um 30 seconds right there will be no questions um if you do two months fee operations to Pine for when you run it you get to see the commands that it spits out over the top commands are the facts that it's the shell commands that it executes for the facts the bottom commands are the ones that the output of the operation that I ran and that shows you yep exactly what it is it's doing if you use three minus fees it actually sits there and shows you the command the response from those commands coming back this is great for debugging any Lessons Learned yes time up talks more properly uh no questions this time we're really done I apologize sorry honestly pretty good speed run

Channel: PyCon AU

Views: 910

Keywords: DarylTester, pyconau, pyconau_2023

Id: 46iD5SActGs

Length: 31min 41sec (1901 seconds)

Published: Thu Aug 24 2023