Deploying ChatOps with Ansible Tower by Red Hat and GitHub Enterprise

good afternoon so I'll go ahead and reduce myself and let them do his my name is Lee Foust I'm a senior Solutions Architect for github I've been with github for almost two years now before that I actually ran my own business doing business transformation we used github whenever we went out and worked with customers it was a lot of best practices a lot of what we now call DevOps I'm also a former Red Hatter so I worked at Red Hat from 2006 to 2009 so I kind of feel like this is like my high school reunion you know I got to come back after 10 years I got to come back to see Red Hat again there we go so don't Silva principal product manager for ansible I handle technical integrations between ansible core and ask the tower with other ISPs in our ecosystem so the conversation that we're gonna have today is sort of threefold so there's going to be a number of different products in the mix so when I actually go out and I work with customers and partners one of the things that we're normally doing is we're talking about best practices and this is one of the things that has really gotten a lot of attention to both our customers and our partners and when I go out and I talk to them I talk to them that we've gotten pretty good about application automation so developers have gotten pretty good at continuously building their code see ICD build a package and then once the package is there it just kind of stalls on the infrastructure side we've gotten pretty good at some infrastructure automation so we've got tools like ansible ansible tower we've now we're using virtualization we're using containers we're using ways of being able to all made all the way to the point that we have a show what happens is is from the show and from the package is sort of like our last mile and how do we bridge that gap one of the things when I talk to CIOs and CTOs is that gap can be very very wide or that gap can be very very close that gap is not a technical gap that gap is a cultural gap and the two pieces that we use inside of github to bring that cultural gap together is through transparency and through collaboration and when we talk about that from a github perspective it's a very open source mentality so when you're actually working inside of your own companies one of the things that PayPal wrote a book around us actually something called inter sourcing and inter sourcing is basically taking the best of open-source workflows and applying them to your own business internally so a lot of the conversations that we usually go into talk to customers about our what these best practices actually look like so what does collaboration actually look like from our from our customer's perspective so this is a traditional sprint so those of you who are not developers this is gonna look like something new the place where the operations people come into is at the very end so this is both sides of those it doesn't go that the slide before it doesn't go from one side and then go off on the other what we're seeing today is both sides are developing in parallel and they end up meeting at the top of that bridge so when we talk about what the Sprint cycle looks like is usually a developer has an issue a bug something that they're working on and they end up having to take on that task and they end up being put out on an island the other thing is is we take a task and we always think of that task as something that will always make it into production but that's not always the case as developers we're not perfect we break things we try different things we try different frameworks we find that one framework based on the way that our security system is set up would never work when it goes into production we find out that the way that we want to render a particular sequel query is going to run a huge CPU spike on the servers so we have to go back and we've got to revisit so what we try to do is we try to talk about actually bringing other developers into the conversation because the more people that collaborate early on in the process what happens is we find that where you make less mistakes when we're actually in production so we have this iterative cycle in the middle so we collaborate we do continuous deployments all of those pieces in the open inside the company it's not an easy task I remember the first time I opened up a pull request instead of github now I've been a developer for over 20 years and I respect the developers at github immensely just like a lot of other people inside their own companies there's other people inside of your company that when you get ready to go ahead and show this is my first ansible script that I'm gonna go ahead and write now I want to run this in production it's kind of scary it's like I have no idea if I did this right I read all the books I went to the training class I think that what I'm doing is the right thing but then you have somebody who's been doing ansible for three years four years and also they come in or Michael DeHaan comes in behind you and goes yeah that's completely wrong you should break that out you should do this differently and in that culture of collaboration what happens as we find so many developers when I opened up that first pull request I was afraid that people are just gonna come in and just tell me everything that was wrong I was really shocked to find everybody coming in and just helping me be able to move that code forward helping me get to the next level so they were a lot of things were hey this isn't the way that we format this particular for loop inside of github you should probably do it this way people pushing me and saying hey you should look at this unit test it looks like something that you would probably want to include as part of your pull request so it was very constructive and I got I was very excited and when I worked at Red Hat and worked with Fedora community it felt a lot of those things felt the same so what we're gonna walk through today is actually what that workflow normally looks like and we're gonna focus in primarily just on the ansible side and more grants start off going in looking at some ansible code we're gonna look at what that looks like inside of github and then we're gonna go ahead and then Dylan's gonna take it from there take us into tower and then we're videos we're gonna come back in and look at what that looks like inside of a chat client all the way looking at the entire flow so let me go over into my browser so here you can see how many people here use github yes we're actually getting the message out there that's awesome so this is Luke's contribution graph poor Luke is a very lonely developer he is not he has no green dots so he has no contribution so he's looking for something to do so he recognizes down the bottom there's actually part of this organization this organization is our operations organization so inside of here he sees a repository they has access to and he's all excited because he just came back from an table training so it's time to go ahead and get his feet wet so as he goes into the repository repositories inside of github are a combination for us of code it's our issues it's our pull requests and then we have some metrics there's also other things as an admin that you can go in that I'm going to show you guys after we do some of the chat op stuff how all these pieces are actually tied together so as he goes in he's looking at some of this code and he remembers that there was an issue that somebody had opened up and here inside of the issues that we have ok so he's got an issue he's looking through he can see what sort of needs to be done so what we're gonna do is we're gonna go back into the code and one of the things I'm not sure I actually showed Dylan this nice little trick if you're not aware of this on github.com or github enterprise if you hit the T key you get a fuzzy search and this fuzzy search I can start piping anything that I want if I remember correctly it was rolls so when I type rolls what will happen is is underneath it's kind of hard to see on the screen but everything that is rolls it's now showing me all the roles locations inside of this code it's all in bold so underneath the roles we had a yeah so inside of here what we need to do is we know that we're actually getting ready to roll out a new version of this application so the new version of the application is actually going to be version 1.1 so from a github perspective the first rule what if we go back to that other long diagram the first thing that we need to do is we need to create a branch so what I'm going to do is I'm going to go ahead and say change version number see I'll hold on Luke does not have permissions they log back in sign back in is yeah this is my alter ego so I'm gonna go ahead and create a new branch and now off of this branch we're gonna go back into our roles and that was so inside of here I can see that I'm on this branch so what I'm going to do is I'm just gonna edit this right here now normally inside of I get in github you would actually clone the repository you would create the branch locally you would make the change test it locally make sure everything's working and then you go ahead and do a push but we're gonna bypass some of that real quick here so what I'm gonna do is I'm gonna change this to version 1.1 and now what I'm gonna do is I'm gonna say I'm committing directly to the branch that I want so update version number for production and I'm going to commit the change now one of the things when I go out and I talk to people is they get stuck in some old workflows whether that be CVS subversion perforce other tools in git branches are very lightweight what a branch is is is basically just a sandbox that's only going to measure the deltas of all the files that change not actually touching and changing the entire file so creating branches is not a problem inside of yet we actually recommend actually creating more branches and part of the reason why we like to create branches is because we can now off of this branch create what we call a pull request how many people here use pull requests alright so those of you who don't use pull requests this is why we create pull requests now what happens is is I now have a place that I can actually start a collaboration flow so Luke is so even though I'm logged in as Chewbacca if I was Luke I could come in here and I could actually add mention the ansible experts team and what does that mention will do is it will actually send a notification to everybody who is in that team that they've been invited to this conversation so now they kind of know hey somebody's actually asking for my help this is sort of like a little forum but the great thing is is it's going to give them context because inside of here will actually be the commit and the changes that I actually have corresponded to this change so you can go in you can preview one of the things that we would normally do like if you look at some really good open source projects or even what we do inside of github one of the things that we actually recommend to customers is usually it's not going to be this short so other things that we can do is we could go in here so let's say this was some major functionality so what I could do is I could actually have additional tasks and I could say write unit tests push to push to dev write some Docs and what this will do is this will actually create little check boxes for me so we actually use this as a way to be able to track where we are inside of the state of a particular item especially if we have different people that are responsible for this so even though the infrastructure team may be writing some of this code I may actually be touching many pieces of the infrastructure so let's say I'm writing an ansible script that's actually going to go in and actually modify a storage device or I'm actually upping the memory from 8 gig of memory to 16 gig of memory or maybe I'm going in and I'm changing the virtual CPU size I might need to involve the capacity planning folks I might need to be able to bring in the storage architects other people I could actually assign them other tasks and when they actually check off and say yeah this script actually looks pretty good we've got the capacity everything looks great this is actually gonna do what we needed to do they can just do a little check mark inside of there so that way we know that we've actually gotten a review from them that it's ok to move forward so now when I create this pull request what we'll see is we'll actually see a commit underneath the other thing that we're actually seeing underneath is if you saw there very quickly is it actually ran a very simple check so what happens is is right now we're saying that this branch has no conflicts on master so the reason why we create branches and branches should be short-lived we don't want these branches to live really long like they would in something like subversion so the reason why we want them to be sure lived us so that we can actually merge them back into the codebase so now if I actually look at the changes if we were actually collaborating with other developers you could actually go in here make line-item comments so somebody else can come in and say hey are you sure that this is version 1.1 a or this shouldn't be version 1.2 and then we can actually collaborate on that the great thing is is when we talked about before when we were actually discussing about developers always pushing something into production one of the things that I've seen that can be very valuable for this is let's say I need wanting to go to a newer version of JBoss and there's a new application that's coming from the developers and we know that the memory for the heap actually has to go from four gig to eight gig of memory well for us to go ahead and do the capacity planning and to go and we may have to go to procurement go get new hardware go ahead and upgrade those VMs all of that isn't going to take a day or two so what we can do is we could actually close this pull request knowing that there's other actions that are going to be taken outside of it so it's not churning up developer space project managers aren't going where are we I just closed it out because I'm just saying that my work is complete for right now once all of the other infrastructure stuff is being complete maybe there's a maybe you've tied this into something like ServiceNow or something like track or something like a JIRA Service Desk and those other pieces on the back end once all those pieces are complete you can then correlate those back into this pull request when they are all done you see that they're all done you then come back in and then you open your pull request back up and then you can complete it so there's other times that we may even take a particular pull request you may actually look at and say this is not the way that we want to go forward we don't want to do this at all I can actually go back into the pull request into the conversation and I can just close the pull request if we go ahead and close it the great thing about having a pull request closed is if I just was keeping everything like you hear people talk about documenting everything inside of the code well what happens is if I just go ahead and I delete the branch I lose all the history all of the changes that took place inside of here I actually have the historic record inside of a pull request you'll actually get an option down the road that if all of a sudden you went down a spike and you were trying something new and then all sudden there were some framework some things that we're missing you could actually go in there would actually be a button that would say restore branch it would actually go back grab all of the changes that we're taking place repopulate that branch recreate it and put it back into a new pull request and it would automatically link to to pull requests together so you can actually keep historical records inside of your company about why or why not did you move forward with these sets of changes before they ever make it into the infrastructure environment so what I'm gonna do is I'm going to merge this pull request one of the other new features that we have as well inside of github I'm actually using github enterprise so if you notice up here github get aboard com github Enterprise as a version of github that you can actually install on your own network it's just a virtual appliance and then you also have github.com all of our features from github com ended up coming directly and to get hub Enterprise it's usually about three months behind so this feature came out in February of this year and now it's in our latest Enterprise Edition so if you're actually doing an integration branch workflow which we could talk about later what you can do is when you actually want to move that into production and you want a very clean history inside of your production branch or your master branch you can actually squash all the commits so basically you have one commit that basically has all of the changes inside of it instead of having all the historical pieces behind it so I'm just going to go ahead and confirm this merge and now that change if we go back in and look at app deploy and there we're now 1.1 turn it over done so now from this point we're gonna jump over to an smith tower before I talk about ansible tower who here knows what ansible core is or uses ansible how many of you keep your hands up use ansible tower today all right cool so let me think this is just canceled yeah pull out my last pass give me one second folks so from this perspective ansible tower is a system where you can actually centralize and manage your ansible playbooks in your ansible automation it's a framework that sits on top of ansible core and allows you to extend that automation by pulling in your playbooks from things like github or other services out there as well as centralize your inventory management so what I'll show you today is a dynamic inventory that's being pulled from Amazon Web Services and then once you have your projects defined and where your playbooks are coming from and then the inventories that you're going to be targeting against then you create job templates and I'll show you the job template that we're going to use to deploy that version 1.1 application on a couple of hosts in Amazon and then we'll talk about how the process for DevOps is achieved using a framework like ansible tower so without further ado let me all right so when you log into tower the first thing you see are the the four bits that make up ansible tower you have your projects your projects are what we call the endpoints for all of your playbooks so you can see I have three projects here I have a baseline project that's used to baseline systems so if you think from an Operations perspective you have a common role that does your baselining of an OS after the system is first spun up or a cloud image comes online then I have some other example play books to play with and then for here in Red Hat summit we created this project you can see that we're referencing that operations repository ansible play books and then you can see that you can specify a specific SCM branch that you want to check out from since we push the master we're just going to be pulling direct from master and then from a credential standpoint you can actually import credentials directly into tower and they're encrypted so when you provide access to teams within your organization to utilize this project you're giving them indirect access to this credential without them having full fledged access to know what that private key is specifically or a username and pass and then you have some options associated to this where you can clean out a repository each time that an update occurs so let's say that if you go on to the disk of the system and you make changes to the play books on the disk of the tower system this would clean up those changes you make and then you can also do an update on launch for the sake of the demo I'm not doing update on launch to show you how to do an update so to pull in the latest change that was just merged I'm going to hit this button here and then you're gonna see actually what tower is doing is calling ansible to do a get pull from github and update and that job is now completed we can see that everything's okay because the check passed we expect it to work so now inventories when you have an inventory it's as simple as defining a static inventory so you create a host define a hostname and then you can create groups for those house and place them in that or you can actually create a dynamic inventory so I'm pulling direct from Amazon at the moment and you can see I have two hosts that I already have pre-populated for my Deb environment so when you look into these hosts you can see some variable information is automatically pulled in that's associated to each instance and these variables can be utilized in playbook runs for anything that might be associated to the host specifically that you want to do in the case of of our playbooks we don't we're not using any of these but they're available for you to utilize we also create groupings automatically based on some tags that are associated to these instances within Amazon may be the keys so you can see I have a key called DaSilva and these are a grouping and of itself and then you can also do it based on security groups all of these allow you to do more extensive automation based around the groupings that Amazon itself applies you can also create your own groups as well by just adding another group so I want to create a group called testing and then let's say that I had a playbook that was targeting that I would just add each one of these hosts to this testing group by just copying it and moving it over to testing and it's as easy as that so this is kind of your your inventory control within Tower once again those users outside of operation let's say those deaf individuals that you wanted to give access to this inventory you could grant them access to it to read it and utilize the hosts that are in that inventory without giving them edit access or you know accidentally deleting a host or doing something that they shouldn't be doing effectively so now we have our projects define our inventory defined you have your job templates a job template is basically ansible - playbook the name of the PlayBook that you're using and inventory you're targeting how does that look as far as we're concerned so today we're concerned about this deploy app so I created this ahead of time and you can see I gave it a descriptive name whatever I wanted to call it it is a run because I'm actually applying this playbook you can do check mode as well scan is a special type of job to gather details about the hosts under inventory we're not going to look at that today so you can see the inventory that I'm targeting which is that Red Hat summit demo the project that I'm pulling a playbook from in the actual playbook and we're gonna use my private key anybody want to see my private key cool thing as I mentioned with any credential that lives within Tower nobody has access to it even the person that put it in it's fully encrypted so you can grant access now to systems to developers that you wouldn't been able to in the past you can give them that access to do deployments to do tasks that they wouldn't be able to in the past so then from there it's as easy as clicking this launch button so now we're actually going through the process of that playbook that's defined and we're going to deploy version 1.1 of the application so while this is running in the background let's actually pull up the app and see if I can catch it in time so we're gonna go to 8080 web app and actually already updated I wasn't quick enough but what you would have seen is that this is version 1.1 it actually would have said 1.0 but I missed it in time because tower moved that quick that's how how fast ansible moves with doing a deployment and you can see within tower we actually clearly define the tasks within the play and then the event details and the results of each individual task has some info associated to it another cool feature that comes along with it as well as while you're watching a job you can actually look at the standardout in real time as it's running against the system's looks familiar right for those of you that use ansible core all of that is fully available in tower and finally this job section is a way to look at previous jobs that have run on the environment and you can see you know this one specifically I had some issues with my job when I was writing my playbook we found out in development that actually the the version number of the app we were deploying wasn't correct we didn't specify the V and it it helps us kind of understand what might be going wrong with writing play books it's a lot cleaner than looking at standard out from shell or using other systems to run your ansible playbook the final thing that comes along with this and this and this specific feature is what helps enable the chat ops approach is our API our API for ansible tower is fully browsable most features that are within the UI of ansible tower are actually driven by the API in the backend so from our perspective any integration that you want to do to help make ansible the central automation tool of your whole environment with other systems is leveraging ansible Towers API this is the final puzzle piece effectively to connect all of your services together so what we're doing with github and their API space compared to our API space and other features that my team is working directly on with other integrators is leveraging this Tower API so with that now we can move on to the chat up story and take a look at some cool things there so let me switch over so first thing first how many people here know what chat ops is Wow I can tell you I probably did my first chat ops talk about 18 months ago and like one hand went up so chat ups from a github perspective is we actually have been doing chat up since 2011 and chat ups for us there's a gentleman who works at github his name his github handle is technical pickles his real name is Josh Nichols he was actually the creator of Hugh bot and Hugh bot is just a bot that sits inside of any chat there's chat adapters for Skype there's ones for HipChat ones for flow doc there's ones for the one that we're looking at here is rocket chat which is an open source slack competitor you can also run it inside of slack so from a chat perspective one of the things that they realized is as the git infrastructure for github was getting very big what they realized is trying to maintain things like username and passwords and trying to figure out who was logging into what machines and what actions were they doing inside of each one was hard so what they is they said well the most secure way for us to be able to get to any machine is by using a public/private keys so they said yeah this will work so for our 22 operations people will just have them create all their keys we'll go ahead and upload all those keys and then when they forget their keys then or they delete their keys or they upload or they upgrade their machine then what we'll do is we'll have to recreate all those keys and everything will all get lost cuz we won't know whose keys are where so what they did is they decided well what if we just had one user with one key and what we could do is we could basically write a series of scripts that would allow us from one chat bot to be able to manage our entire infrastructure so there are things like the one that is the most common one that we give us an example is I guess it was about a year a little over a year ago github was hit with its largest denial of service attack that ever happened to any company in the u.s. at least publicly that they would admit that it happened one of the things that immediately kicked off is our bot was actually sitting inside the infrastructure and was actually monitoring the activity that was actually going on and there were certain metrics through our CDN provider that actually came back and through their API we could get flags back from different colors different states of being able to say are we actually under a denial of service attack so the person in operations who is actually one of our security gurus was actually about to go give a presentation in Paris when he got an alert on his phone from the bots that said we're about to be under attack not that we're under attack but it looks like we're about to be under attack he logs into chat from his phone in the back of a cab on the way to his talk tells Hugh about to go off and prepare for the attack Hugh bot automatically goes ahead sets up a series of honey pots automatically starts rerouting all the traffic all the honey pots start returning 404 and 500 so looks like the attack is actually winning in the meantime it automatically redirects all of the external load balancers so start pushing through a separate data center so all of our users who are actually going to other locations for the website would be getting to hundreds and then once we figured out exactly where the IP range where it was I where the attack was actually coming from then we were routed all of those attackers directly to the honey pots they thought they were winning they thought github was down the entire time github status was a green the entire time through so the BOD itself doesn't have to be something that is always interactive it can actually somewhat be proactive for a lot of different activities so the one that we have set up right here is actually when we actually open up the pull request one of the things that we realize is you're not gonna want to sit in tower all day long and just sit there and just watch when people are doing things inside of Tower to be able to figure out what's going on it's the same thing inside of github you're not going to sit there as soon as you open up your pool requests and just sit there and watch it in front of your monitor and wait for somebody to come give you a comment you're gonna go work somewhere else a lot of times we work inside of our email well when we work inside of our email we get so many other things inside of our email that a lot of things get lost so for us ours is inside of chats we actually have chat rooms that are dedicated just to the bots and then we have BOTS listening in each one and then we have certain secured rooms that are only available where people inside of operations can actually do production type deployments we actually run all kinds of other BOTS inside of github our entire sales process is completely chat-up enabled so when we want to go ahead and we've got somebody who just comes into Salesforce and they come in and it's a lead that's been generated from our website we see that a lead has come in and it actually shows up inside of slack for us and then what happens is is we can actually type Hugh bot recon and then we give the domain name of the actual customer from the domain that actually came in from we can actually look are they already an existing customer we can look do they have any open tickets how many times have they come to the website what's a Google Analytics on that customer what partners do we have that are already registered with that particular customer so we automatically have all this information that we can automatically gain because we're using all these api's into all these other systems so this is an example of some inbound ones that we actually setup from github that when we actually look at something called web hooks inside of github so here I have a web hook set up to be able to say anytime there are certain events that actually take place inside of github I want you to route the JSON payload for that web hook to the bottom and then the bot takes that JSON payload and transforms it into something that we can actually read inside of chat so that's one way you can actually send data in and the bot can actually listen and send it in a chat well normally when you get that kind of data you're then going to want to react to something that has actually come inbound so using the great api's from tower what I can do is I can actually go to Hue bot I could say hey Hugh bot give me the tower inventories that we currently have so as we wait for a chat to respond so Hugh bot actually you saw how quick that was Hugh bot actually went to the tower instance actually use the API actually got back that there are three inventories so there's an inventory in each inventory has a number this is actually the ID of the inventory from the API so then if I want to ask Hugh bot I can then say Hugh bot tower inventory and then I can pass in the number for the one that I actually want to look at and here I can actually go in and here you can see that this is the inventory for eight I can actually see all of the different inventory items inside of there now you can imagine now I've got the IP address of that inventory I've actually got the Amazon I've actually got the ami ID so now there are other plugins that you can use the Hugh bot is actually based on JavaScript so it's actually based on node J s so there's all kinds of other plugins that you can have there's an AWS plug-in so I could now have Hugh bot I could actually drill in and ask you bot and say hey go ahead and tell me more about that Amazon ID that's sitting there get that ID back I could then go ahead and say hey give me the last hundred lines of the state of the of the standardout log from that particular machine and they can pipe it and send everything back into there so as you go through this process you end up being able to do a lot of work directly from inside of chat without actually having to open up a window now this is sort of one-to-one relationship now imagine just like what Dylan showed you with the ansible script where he actually went out and provision two machines or updated two machines at once off of these inventories I could actually have the bot actually orchestrate many activities at once at once across many different services so you could query a bunch of different places and then based on the responses that come back go into the next step and actually perform a number of actions for that so those are sort of the two different levels there's lots of things inside of Hugh bot that you can actually go out there's a number of other were not particular Hugh bot is completely open-source you can take that and you can actually do whatever you want with it you can actually build your own for fun I actually wrote a plug-in for ActiveMQ and another one for RabbitMQ so what you could actually do is you could actually use Hugh bot to actually listen for things off of a queueing system so if you've got something on the back end like a mainframe or something that's actually listening to queues I could actually get data off of that so there's all kinds of different ways that you guys can actually do that so from there what I'm to do is anything to add you know I would say from a tower perspective to what what we're working on for integrations with in chat ops for all of the different chat services is every single item that lives within Tower so updating your projects creating inventories doing job template dates and pushing jobs out to your systems will eventually all go through chat ops services in some form or fashion what I envision in the future as well is from within chat systems such as rocket chat you could actually look at your job status without actually having to leave and then if you want to dive into more details you can click on a link for each individual job and it will take you to tower so you can look at further details from there because it helps centralize it because I know myself just like Lee said I don't want to jump it back to tower every single time I live in chat all day long so why not keep it central to something that I love working out of and that's the whole bit about chat ops and because ansible is a true automation engine is using that bot to connect to tower as well as the chat system reaching out to all of the other services it helps centralize everything and keep it in one nice package to work along with so from my standpoint I think chat Ops is one of the most exciting things that we can work with today yeah and just to piggyback on that is also for those of you that have to deal with like SAS 70 audits this is now a single place that you can actually have a record of not only who did what action but you have a historical record for as long as you want to keep it so now you can say hey I want to know what command was run on that particular server so if you're keeping track of and you make people have to sort of walk through some of these steps you know which scripts are gonna run that are actually going to modify machines on the back end because they go through the other thing is is now every machine inside the system you don't have to worry about somebody coming in said okay so who has access to these machines nobody what do you mean nobody has access to any of these machines no we do everything through chat there's one bot who has access to all of our machines nobody can SSH into them so then that actually eliminates a lot of the reports a lot of things that you have to do in some of the security audits that you have to do because you're minimizing the amount of touch points inside of your infrastructure which is really critical especially when you start talking about imagine what happens we start getting into docker sprawl and you start talking about one service per container and all sudden you've got ten thousand containers we thought a thousand VMs was bad now we're gonna have ten thousand containers that are running and people who are stopping and pulling and restarting and making changes who did that who pulled down the change who did the put who did the change and did a push back up those are the kinds of things that you can now do I'll do through auditing and being able to see those kinds of things through chat and you know to finalize that out that's the beauty of using tower along with this is because it as you guys saw every single action that occurs via a playbook is logged so you have a way to generate those reports even further and it's granular enough for SAS 70 audits I myself my previous role that's how we did it for going into our IPO launch so it's it's definitely a system that can be utilized to do that kind of stuff great so any questions yes sure it does not show who checked it so one of the things that we try to talk about inside of github is again through that transparency is there also has to be a trust factor so when somebody actually checks the box it is a request yeah it is a request to be able to say edited by so there are some things that we're looking at about being able to create more traceability inside of github so definitely noted it's it's a common request so the the piece that we're trying to look at doing is you know a lot of the things that we run into is we want to try to keep us clean of a UI as possible and then if somebody unchecks it to that we then track who unchecked it and then if somebody else comes back in and checked it do I keep a historical log or do I keep those as comments on the side or do I keep it as a hidden div and show it only when I need to but one of the things we actually have a very large customer that what happens is as we can actually trace the action on the page it is actually stored in the database we just don't represent it inside the UI so if you did need to be able to say hey I don't know why somebody checked us you could actually go back and and query the database and actually get it who did that action yes yeah that's a I remember when I first started with github I always brought up with everybody I remember my first open-source pull request that I did and it was for bootstrap and I sat there and I made a whole bunch of changes for a Twitter bootstrap and I went ahead and I submitted my pull request thinking the exact same thing thinking I fixed it just go ahead I'm waiting for somebody to merge and what happened was all of a sudden this huge conversation took off about architectural 'hey hey we really like what he did maybe other components could take advantage of this as well next thing i know also that's like three days later and all sudden i'm scrolling trying to figure out where the end is and also i get down to the bottom and it's closed and i'm like what do you mean it's closed and what i found out is they ended up creating other pull requests for each one of the components I made an architectural change higher up but they took my idea and down the bottom there all these plus ones all these thumbs-up yes we should move forward with this but it was the conversation that was important not to change so what happens is is a lot of times from a coding perspective we expect whatever we write to be merged and the change that we end up see inside of a lot of open source projects from a github perspective is the change that we're trying to make is a suggestion not the fix so when you go into that mindset then what happens is the pour request is the start of the suggestion that suggestion may or may not get merged but it's the conversation that lives forever so now if you ever go back and you say hey why didn't we merge that thing in Oh remember that thirteenth comment down that said if we go ahead and do this that we know that sha-1 is about to be phased out in shop 2's coming this would have limited us to have to support sha-1 for the next five years if we would have merges into master so then what you can do is we do in github a lot we do what's called cross-linking so inside of a pull request if you just use the pound symbol you can see the issue so you can see other pull requests and then you see the link inside of the pull request you actually see a line in there that says this is related and then you can actually do that traceability in that linking and we have a lot of customers and partners that have done some really interesting integrations by being able to trace that through from a requirements perspective so or create a release notes at the end of a particular version so totally understand one of the things that normally would have started that conversation is there would have been an issue that issue would have been open somebody would have gone ahead and made a code fix that directly applies to the issue one of the other things that if you're not aware of is if you say closes and then use the pound symbol or say fixes and the pound symbol when that pull request is merged it'll automatically close out the issue on your behalf so that is usually the path is from an issue into a pull request but the key thing to remember at the suggestion so we want people to make the smallest change possible to create a pull request start a conversation and then you started a collaborative workflow that you go ahead and you start having your changes one of the things that I love is it's not on the ansible side but like from Java development perspective is this is awesome when you start talking about wanting to see your developers follow a test-driven development process because what you do is you say I don't want you to start a pull request when the implementation is done I want the pull request to start when you create your BDD or your tests or your unit tests because now what happens is I can actually go back in and say yeah this looks good I like where this is headed let's go ahead let's start seeing the commits for the implementation and let's start seeing the build start to succeed the other thing is is I can't tell you how many project managers that I've had to teach that when they go ahead and they open up a pull request for the first time and see I comes back and says that the build fail and they're going horrible developer yep see that first commit can't even commit one thing it's already broken I'm like no they submitted a unit test that doesn't have an implementation with it yet and that's actually what we want to see we would actually like to see a couple of fails early on in the process so when you actually go through that process that that conversation leads also to pushing more of the fix of the code at the beginning of the process we have actually done some studies that we've been able to reduce production downtime by over 80% and some of our customers because of using pull requests that fix is actually being found because many people are looking at it we've all done we've all done code reviews and I've I'm a bad developer I will admit it so I was that person on a Friday afternoon at the end of a sprint and it's 3 o'clock in the afternoon my wife's already texted me 14 times what do you know when I'm gonna be home because she wants to be out on the boat or because we've got a tailgate and we gotta go get our food for tomorrow and I'm sitting there and they're going hey Lee what do you think about that like yeah that looks great you know so in the middle of a code review we don't always take it seriously and we always find things we also find them at the end of the process so then you get a project manager or somebody that says is it really bad enough do you think we could go ahead and go onto the next sprint we could go ahead and just like QA see if it's really bad so when we open up that pull request early on in a sprint cycle or at the beginning of a development cycle what happens is we now have more time to be able to fix something in that process rather than waiting until the very end and then trying to push it through because we have a deadline to meet does that help great awesome other questions no other questions I know it's Thursday afternoon everybody's ready to go home one back yeah absolutely again it's that trust factor right so not everybody would have access to that room and that room you would have so like the way that Hugh bot works you don't have one Hugh bot for all of chat so you have one Hugh bot per room and then that Hugh bot would only have certain plugins enabled so maybe depending on how you organise your plugins maybe the deployment plugins are only available in a certain ops room and maybe there's other ones that be able are allowed to be able to get the logs those logs are in another room so those two different things one who's allowed to be able to change infrastructure and another one who's able to read the infrastructure may actually be two different rooms and it really depends on how that organization actually flows but we've we've managed that because I can tell you one of the things I can't do it github they will refuse to allow me to push to production I don't know why maybe it's because I go around telling everybody I'm a bad developer so but I'm also in some of the tools like in slack and rocket chat some of the things are very granular the security is very granular so I know in rocket chat you can actually decline people from actually running bot commands so if they type in a bot command it actually gets ignored with a little nasty gram that says you do not have permission and they can actually do it off of a regular expression so if you're a user and you don't have the ability to do a deploy action what will happen is it actually sends a nasty gram and it gets long that you tried to do a deploy and then it's like hey did you know you weren't allowed to do a deploy and then you can try to find out somebody's actually trying to be malicious or you could find out if it was just a training thing so different ways of being able to have that scenario but that's a great question and I guess from the the tower standpoint we have our back built into it so each individual you know I had mentioned that you can grant them access to projects you can grant them access to inventories and job templates and credentials everything is is in a explicit grant in terms of the permission so from that perspective the one-to-one mapping between Hugh bot in this case and tower is that user would have access within tower to do those commands and then you see the logging happen between the systems in this case the way you could extend it even further is that the individual users logged into a chat system will have a one-to-one mapping of a user within Tower and then have some communication go back and forth via the bot that does a handshake so to speak those are some things that were we're thinking about how we can actually implement that because you want to go even further with that auditability trail so to speak yeah so we actually have it in slack that we you actually had them extend slack for github for us that we can actually load our github personal access tokens and then we can actually pass in like Dylan was saying as an encrypted we actually have a variable that we can actually use the bot we can actually pass our token in to be able to get access to certain pieces of the infrastructure that requires some sort of identity to validate you are who you say you are yeah just think of the bottom mostly as your your API coordinator really at that rate yeah other questions awesome well thank you all very much appreciate it

Channel: Red Hat Summit

Views: 782

Rating: 5 out of 5

Keywords: Red Hat, github, devops, ansible, ansible tower, jenkins, automation

Id: qkToa_J7ssw

Length: 56min 4sec (3364 seconds)

Published: Thu Oct 20 2016