SaltStack is Not Configuration Management (It's So Much More)

my name is drew Malone I'm with cloud era I am the DevOps guy I've been introduced as the DevOps guy every time I go and meet someone so I should start putting it on the business card so what we're here for today is salt is not configuration management it's so much more than that quick background on myself again with cloud era I'm on the internal tool systems and tooling team in cloud era all the in-house applications I help streamline them take care of them curate them if you were here at last year's conference you might have seen my talk on salt cloud entitled saw the cloud and make it rain trying to keep up with the interesting tag lines and was fortunate enough to be here for the first salt coffe where I was able to sit for the exam and got myself a single-digit certain number so just to be sure so you've got about about 30 more seconds before you can walk out of here without hurting my feelings this is the slide that's gonna let you make that decision if you're new here new to salt and you're overwhelmed by the docs we're here to help if you're a veteran to salt you already know all the tricks you're tired of them you want to learn a few new things hopefully you're still in the right place if you've really just wondering why you're still here and you just feel an obligation to hang to the end of the conference I promise it's gonna be quick and in trying to make it entertaining so it'll be worth your time stick with it brief outline of what we're going to be doing today we'll go over a quick history a story of you a story of me a story of how we got here we'll then go over a quick crash course an inventory if you will of the various components and salt that you may or may not have heard of with a little quick description having done that we've got our box full of LEGO pieces and we'll show you what kind of capabilities you can deliver when you put them together in in interesting ways once we're done with that I'll show you all the things that I didn't have time to go over give you a little bit of homework you can read up on them later on and at the end we can all congratulate each other for us making it through another conference so a story remember the first day on the job a lot of machines that's us aging in fat fingering config files fixing miss misconfigured services three of the machines work two of them don't know know why it's not really working out let's try to automate a few things because we're getting more machines I have never once been told we need you to deliver fewer services it's only more services so we're starting to script a few things we're starting to get consistent results you might have a playbook if you're really good you might have a git repo where the team shares those scripts once you get the feel for that we move on we start actually using we're using config management now we're getting comfortable with salt the team likes it management either doesn't know it doesn't care or you actually have buy-in we've got automated deploys it doesn't matter how many services we're delivering now because we've automated it all things are consistent they're reliably up things are great congratulations you've won y'all can retire now you made it successfully using configuration management to manage your services deliver business capabilities so someone in the audience feel free to complete this sentence what's the reward for good work more good work that's right y'all so good so you're delivering all these great capabilities y'all are really good at what you do you need you to do more we need you to put together these completely unrelated systems we need them working together in harmony we need them delivering new capabilities we need you building not only more of these systems but more complex systems than this one you just finished oh and those are going to be a little more complex as well also we're going to have to have them delivering capabilities in real time we can't do batch processing we're not going to be working with customer requests we need this thing working up all the time and we needs to be trivial to deploy something even if it's not related to any other any other system great I can't use a config management system to deliver all those things I got to learn more tool sets now I got to go and find out what new tool sets are gonna help it deliver those capabilities as it turns out you already have those tool sets if you're using salt like I said this is so much more than config management we've got a whole set of components that you might not even know you have you don't even have to install anything extra so my intention here is to show you that they're not just interesting tools that you can look into out of curiosity these are very powerful tools when assembled in interesting arrangements you can deliver capabilities that frankly I have not seen any software open or paid deliver capabilities like this before so the last point I want to make before we go on with the inventory again salt is not just configuration management salt is a parallel remote execution engine built on top of an event message bus upon which you can build a configuration management capability but you need to realize that event bus is the central nervous system into your data center you can you can build so many more capabilities on top of this when you start to see what these components are able to do so I'll give you about 5 10 seconds to read this it turns out it's really difficult to make a good-looking yet organized list of components a word cloud is confusing and putting a grid in the table looks like a table with a grid so it's an eyesore so with a show of hands can I get the people who are familiar at least conversationally with every item on this slide amazing you guys are one of the 10 lucky 10,000 today thank you for coming out we're going to go over each one of these so salt reactor like I said you have a an event bus that you can use that is the central nervous system into your data center what if you had some process that could monitor these things that could look for an event on the bus and when it sees an event on the bus fire off some some ability run some script fire some API call this is what reactor does for you you write your salt states you map them to events on the event bus we'll get more into the application of this a little bit later so cloud lets you codify your footprint in the cloud so you you can write a simple yeah Mel config file to describe what OS version what flavor what size instance and you get to pick your cloud provider Rackspace digitalocean easy to you name it they've written a parade driver for it it's amazing the amount of work that's that's gone into salt cloud it makes a trivial to deploy complex web systems up in any cloud of your choice to include VMware and local hypervisors Salt API is a favorite for especially something like continuous integration CI CD platforms this is your programmatic access to everything salt with a proper salt API set up anything you can do on the command line you can do with the API and now remember that's in just a quick HTTP POST call so anything that you can write that can send some HTTP POST to your salt api system is now your entry point into orchestrating more complex workflows the salt mine is a small I don't want to call it at a data base but it's a small storage of information provided by the minions handed over to the master and you get to define there's there's a handful of pre-built data that can be sent in from the minions but you can also define any interesting facts about your minions and have them send it over to the master to be inside the salt line once it's there a simple query into the salt mine executed by any other minion in your system let you get more information out of your deployment salt orchestrate amazing if you're familiar with state files with SLS files salt runs those are scoped to single machines you can what you can run complex installation and configurations on a single machine orchestrate lets you take those atomic work units and go one level higher so that now the scope of work is no longer a single machine but is every machine in your data center to to bring it to a point as a trivial example suppose that the database server needs to come alive and be ready before you're going to stand up the web server farm before you're going to stand up your reverse proxy that's going to sit in front handle all the traffic orchestrate is the capability that lets you define that order of operations anybody here ever get pushback because they don't want another minion in their data center I understand it's so this is your answer so you don't want don't can't don't want or can't have a minion on your system no more agents fine let's use salt ssh still surprisingly fast even though it turns the transport layer from the 0 mq message bus over into ssh you lose a few of the capabilities but if you want configuration management and you have staunch upper human management that resists the installation of any more minions this is your break and enter team salt vert the primordial phase of what salt has become I'm leaning on some of the salt members to correct me if I'm wrong I believe salt vert is the first iteration of salt because Tom wanted something in database lists way of controlling hypervisors every everything out there was was too complex you just wanted a simple way to manage a local hypervisor so what's offered does is it your laptop into a data center if you have a hypervisor you can very easily spin up a bunch of VMs on there without having to know anything about your underlying mechanism engines are interesting it's kind of it's one of those things where when you first see it you think I'm not really sure what I would use that for but when you find the use case for an engine absolutely nothing else would do instead this is a process that runs alongside your salt master or minion I believe and has access to everything salt has access to as well so if you want to write your own process that can attach to the event bus and do whatever you need to monitor the events send out metrics catch API requests and execute some function as a result anything you need to you can very easily define all the configs and parameters in the same config setup as you do your master or a minion very easy to get stood up beacons the close relative of the engine there your boots on the ground they are the ones that live out on the minions and they also tied to the event bus and you can have them well do anything you can have them map an event on the event bus to some action or if you're tired of the event bus you can step out of that and they are your exposure to the real world as a quick example you can use something like the eye notify beacon to monitor some config file that's very important to you a beacon can fire off an event that says this file has been modified and you can and you can configure your salt deployment to update that config file immediately we'll get into that a little bit later in the in the how-to just in case those components are not enough for you you can customize it salt is intensely hackable it is very easy to get started you can write your own custom modules you write your own custom grains you can implement your own pillars but you it's it's incredible the things that you can do if someone hasn't already written it which is surprising but if it's not already been implemented you can write your own module and the documentation is actually very good at helping you get started with that so that's our quick inventory of the various components of salt so let's walk through a couple of use cases when we put them together in interesting ways what I was just talking about state enforcement so take our beacons combine it with reactors and you get what I like to call anti cowboy measures anybody have a resident cowboy on the team anybody have a couple of Cowboys on the team so here's how this works we're gonna minimize the impact of that cowboy we have defined our config file it does look like this we shall put the config file out on our systems the cowboy comes along I want to put my buddy in there I've got access to the system I'll just add him in there that'd be great you beacon immediately fires off your I notified beacon says this files changed you need to do something and so when you've configured it properly what salt will do and this is immediately you don't have to wait for your next high state you don't get a notification telling you it's changed this does it for you you do nothing it will immediately turn around and restore that config file restarting the services if necessary it's like the cowboy was never there one of my favorites salt cloud so I understand the pre-conference training all the VMS that we all logged into were all made with salt cloud very easy to to define that define your provider your profile get it set up again the docs are very good again you started then it doesn't have to be a size of 1000 or 100 you can have five guys again even get a gettin together in the office after hours and this is great because not only will it make VMs for you you can specify follow-on automation to set up some development environment or set up some class environment with examples okay so we'll get to something a little more interesting this is a three-tier web app deployed with salt cloud if the text is not too thin or small you can see what this looks like this is a simple gamma file describing the UI a little MongoDB server and an engine X to sit in front of it all in a reverse proxy so we feed that to Saul cloud it rolls it out there instant three to your web app and the last one is I don't have any pictures or photos or anything with this is a story I was told when salt was brought in just as a requirement a large company was was was running a bake-off they had to have some vendors come in display some capabilities they needed the capability to run some compute farm it doesn't matter what it was they needed to be able to run it in some cloud and so bringing salt into the conversation was an afterthought and so what they did was they went over the requirements on a Friday and over the weekend what the one salt attack employee managed to pull off over a slow weekend was build this small small simple automated script that would query the costs in all of the available cloud providers come up with what it would cost to run that compute farm in each cloud of their choice automatically picked the one with the smallest cost and then deploy you sell cloud to deploy that farm into the into the cloud provider set up the compute farm and get to work so push a button and you get the most cost-effective result they wound up going with salt in that case portable auto scaling if did anybody take any of the pre-conference training was anyone there for the auto scaling reactors beacons the whole thing excellent this is the this is effectively what that went over so let's see what happens so you have a website it looks great you've got a little reverse proxy because you're responsible users start to look at it the user likes it user posted it on reddit posted it on slash dot shows it to all his friends and then they show up the website is very sad users are very sad tubes they can't see the chasm in a quick manner they have to wait 10 seconds for those cats your site's not popular so what happens here is you have beacons you have the load beacon monitors the load on all of your servers you set your thresholds the beacon says these guys are under way too high of a load we needed to do something beacon fires off a message up to the salt master who then says we need a little bit more horsepower here make me more UI servers and put them in the ant farm fire up the salt cloud add more FR add more servers into your app farm more horsepower faster cats happier users better website again completely hands-free so some people will say we'll just put it in ec2 just put in an AWS they've got on a scaling for free which okay that's cool there's your lotion doesn't have auto scaling Rackspace yeah they have auto scaling but it works different lenôtre they have auto scaling it also works different when you create your own capabilities you don't care where your services live you can put them in anywhere you want avoid vendor lock-in do-it-yourself salt makes it easy so this one is straight from my day-to-day Salt API + reactor never update your production configs again is anybody here familiar with Rheem on the open source project not the mathematician it's okay you don't have to be if you're interested in any sort of well you know what never mind after having gone through the thorium talk we forget it if you're interested in monitoring look into thorium so regardless there's a service it's called Bremen it needs config files naturally we're responsible we manage them with salt we're also extra responsible we manage all of our configs in git we don't want to have to get clone modify our configs git push go to our Iman server take the config files copy them in there even if we do have everything arranged with salt so we just say salt Ramon dot running and then that's enough even if we do that which we have that still is another command we need to run we don't want to do that this is what I want and it's what we have in fact a user takes that Ramon config pushes it up to github which then has a web hook that runs that post remember that salt API post pushes that / - salt API which is then via reactor configured to run our Rieman states which include getting the latest config files out of optionally restarting Ramon if necessary and Bob's your uncle all you did was push the config into github and now the brand-new configs are out there in production no extra work involved so I'm with cloud era we do Hadoop sometimes something that we do often is our our customer operations engineers the guys that solve customers Hadoop problems they need to recreate Hadoop clusters and we have that capability but what we really wanted was to be able to reproduce a carburized Hadoop cluster does anybody have any experience building a heart rate kerberized Hadoop cluster I'm so sorry so Hadoop hard enough Kerberos is hard carburized Hadoop is long long days very long days ok so I'll skip through the laundry list and get straight to the visuals so here's what we decided to come up with we have our own Active Directory isolated Active Directory instance non-production it is just dedicated to the reproduction of customer issues we also have this really clever application that I cannot take credit for writing that we call cluster deploy what this is is our users will fill out a web form they'll click a button and they get a Hadoop cluster so what we did was we added a little tiny checkbox to the bottom of that form that says also Kerberos so what happens on the back end when users click that button is the application will hit our salt API send another post over there telling it someone needs an O you created in Active Directory to avoid getting too far into Windows suffice it to say things need to happen in Active Directory so that your Hadoop cluster can connect to it for Carlos related services so what we have then is our hook into Salt API which contains in the post data is going to be some interesting details about how to implement those things that need to happen Active Directory which then run through a templated PowerShell script god bless PowerShell which then runs on our Active Directory box which creates the OU's creates the user accounts does some delegations and we're done the application is going to send the login details back to the users and they have what they need out of their Active Directory service we don't have to maintain login accounts for our users to log into Active Directory god save us if we had to manage user accounts where our users have login to our Windows Server and manually create ou used to make their kerberized clusters it would be an absolute nightmare this capability has has saved us an incredible amount of time so those are some of the use cases with some of the components I wanted to make sure I didn't go over on here so these is these are just a few of the components that we didn't have time for in here salts Indic is your master of masters I've run salt masters distributed geographically around the planet I mean I've had a master of masters living in one hemisphere and I've had several sub masters living in another hemisphere on the planet this was a couple of years ago when it wasn't even as good as it is today and it was impressive the response time was amazing and how reliable it was as Wow if you need painfully distributed systems look into salts Indic returners are when you run a soft job that return data comes back to the master and you see it on the screen that's nice what if you need auditing what if you need some post-mortems what if you need to do something with the text that's on your screen returners solve that problem for you there's already a lot of stock returners my sequel favorite database of choice and as we said earlier you can build your own return or if that's not enough so if you have any auditing needs or if you need to really dig into the job cache you've got thousands of minions in the screen just isn't big enough to look at the output this is the beginning of where you build a capability if you have a returner that goes into a database you can build a small UI that sits on top of it and you can much more easily analyze the success or failure of your jobs the the SD the salt database is a handy database layer it's it's another one of those components where it serves a niche field where once you find a use case floor you can cannot imagine how you could have done without it and of course the external auth if you've ever just gone through a getting started with salt you may not have realized you're using the stock Pam off module for salt you can use many other authentication systems through the e auth system of salt so if you have a more corporate more rigid enterprise you have a capability to have some better access controls so that if you're anti cowboy mechanisms aren't good enough you can also limit what kind of commands that that guy can run or that team of guys depending on your circumstances so all of that to say salt is way more than configuration management so it's a platform to let you build capabilities that best serve your organization your needs and in my personal experience I've seen and built capabilities with salt that I don't know how else you would make them even if you had paid software and especially with how lightweight and how fast it can be and how customizable it's you can do some amazing things with salt much beyond just managing a configuration file and it's all that it all lives on top of this message bus look more into this the message bus is the central nervous system to your data center you can do a lot with the information that's going across there so this is the end of the main part of the talk and this is where we can open it up to some Q&A if there's any questions rebuttals yes in a word yes so the answer the the question was have I looked into using beacons to deliver a capability similar to nachos is that about right yes absolutely I myself have not done the legwork of doing it but I have seen it done and there's there's absolutely nothing getting in your way from using beacons and with just a couple days ago thorium having been announced if you're interested in a true monitoring capability with taking action and notifications thorium is is is reactor on steroids it's uh it suits exactly the need that you're trying that you're going after so the question with this Kerber has to do cluster capability why a powershell script instead of the Windows capability of modifying Active Directory objects I'm embarrassed to say I didn't look into those abilities now I want to go back and look to see what we could have done I wonder how much time that would have saved me that that was a good question I I need to go back and look at that yes so since we're on the topic so what that powers it does all right so all it does is it makes it it makes a couple of ho use it makes it makes an O you makes a couple service accounts and then it delegates authority of the o you to one of the service accounts that was really cagey to do with PowerShell I'd be really shocked if if that capability was native in Sault windows I mean I would love for it to be there but I understand that's Windows is very hard to instrument III would I will yes all right so if we're out of questions oh we're not out of questions yeah let's go back which database would you mean so the question is how capable is is sdb I've again embarrassed to say I'm not fully first on SDB if anybody else is ok thanks it's a way to all right yes so imagine you've got I'm a huge fan of absoulte API so going from Salt API to like Salt cloud that that chain of of actions is something that really fascinates me being able to build like go with this use case you've got an in-house API you want to deliver some sort of a pat's capability so you've salted the creation of all of these services and all of these features wouldn't it be nice to kind of make that specifically available to some users salt Enterprise and other UI tools notwithstanding you could have a UI that sits on top of your salt master wouldn't even have to be on your master it'd just have to be able to hit the salt API and it would send the data to the to the salt API which then can drop some information on the message bus that's where the engine comes into play when you have an engine you're not limited to just reactor States and salt states you have an entire standalone Python process so you have the full power of Python to do whatever you need to do with that and that includes instrumenting all of the native salt ap is like the cloud client and everything like that so you from there would be able to do whatever is necessary to authenticate to your internal internal or external cloud provider create the VMS and then follow on with bootstrapping them and then adding additional capabilities on to those that's a good question thank you yes enterprise I am totally deferring that one okay so I'm glad to but it's not necessary if you want the windows integration that being said you wanna we can go down the engine path so the so the question is it sounds like you're asking about a little bit more details on how engines work in the context of having to also maintain some Windows servers ok so never mind what window service sure so sure so it's if you want to write a so an engine is a separate Python process that runs alongside your master it has access to everything that salt does at which it includes the event bus it includes all the configurations it can include pillar so when you find yourself saying I really wish I could have the salt master do some capability write yourself an engine have it sitting next to the salt master and just implement that capability in Python yourself so I'll be here a little bit after this but in the meanwhile thanks everybody for making it out if you can spare a feedback I'd appreciate it otherwise thanks again for staying congratulations for making it to the end of the conference

Channel: SaltProject

Views: 1,962

Rating: 5 out of 5

Keywords: salt, saltstack, DevOps, systems administration, remote execution, configuration management, software, systems management, data center automation, cloud orchestration, virtualization management, server provisioning, server management, SaltConf16, Salt API, masterless, SaltStack Beacons, OpenStack

Id: qBtCT96OCfg

Length: 34min 57sec (2097 seconds)

Published: Wed Jul 13 2016