ShowMeCon 2019 07 Confessions of Really Bad SysAdmins Andy Thompson

so let's give it up for me hello thanks everybody I'm gonna stumble over here again with that said before I begin and I want to read a statement that I've prepared so with that said I should be in LA today speaking at another InfoSec comments instead I took today off flew all the way out here to st. Louis and I'm speaking at an impost that conference I did submit this talk on behalf of myself independent of my employer who happens to be here today as a sponsor so I apologize to my employers in advance let's see so with that said the statements today are on my own behalf not on my employer are we getting feedback on that mic all right and we turn this one off so first lesson of today's talk is about microphone management all right can y'all hear me now no feedback still a little bit of feedback all right so with that said you guys see the disclaimer here there is a little bit of bad language it's nothing graphic or anything all right so also it won't hurt my feelings if you walk out this wouldn't be the first and won't be the last so with that said let's see I would like to shake this thang show-me con CFP Review Board for approving this talk and I also apologize for what you all are about to hear also if you're still listening to this then thank you for coming and specifically coming to this talk I'm not gonna be discussing anything new or groundbreaking this is just old stuff this is stuff that you see in your environments to this day so my goal here for today is really to fill the your attention for the allotted time to make you laugh a couple of times and maybe you'll learn a thing or two that's really the goal here today so with that said I've got about a 40 45 minute talk and I've got 63 slides and the slides about are it's about a one gig PowerPoint presentation so with that we'll get started Oh last thing today happens to be my birthday so yeah so I will be at the bar at the end of tonight if anybody wants to buy me a drink alright so let's get started this is talk is confessions of really bad systems administrators we're gonna kind of learn from the mistakes of me and many of my peers so I was mentioning this earlier I'm basically committing like career hari-kari on stage so this either is going to be very successful or I don't know the real title of today's talk is really confessions of a shitty sis admins there's a theme here and hopefully you'll kind of see where we're going here so a little bit about myself I work for cyber-ark software I have the coolest job in the entire company I'm not a sales guy I'm not a tech guy I simply help people so I'm helping partners I'm helping existing customers guys internal into our team I've run what's called our office of programs I really have no clue what it is but my job is generally I just help people for a living I have a degree from the University of Texas in Arlington Texas home of the Mavericks if you're not familiar with UTA we have an undefeated football team since 1976 pretty awesome right you know about 1976 the nation's Bicentennial that also happen to be the same year we got rid of our football program all right I also have a bunch of IT security certifications on big fan of certification I think it's a good way to get your foot in the door in the industry I also run the subreddit for the CISSP subreddit so if you have questions about getting an info SEC this I'm always around to help and answer your questions I am a really active member of the InfoSec community in Dallas Texas so Dallas Fort Worth if you're not familiar has a kick-ass community we between like Dallas hackers association in TX ESG o day all day please school DC - one for Issa is e squared I mean the list literally goes on and on and I'm really happy to say I'm part of this really active and vibrant community I personally am on known as in the community as the travel hacker now I'm not like chris Roberts hacking your plans out of the sky or anything what we've me and my wife do is though we get really really cheap airfare and travel all over the world for like a shoestring budget let's see I went to Australia for 11 days I think we spent $3,500 for that total we did a Easter Island for again nothing nine days for cost like 1,100 and just recently like literally like 48 hours ago I just got back from Liberia Costa Rica we spent five days there and including food airfare lodging at this resort I think we pay 700 bucks for the four of us so if you want to learn more about travel hacking I'll be more than happy to talk to you about that later but more than anything beyond all that I was a former shitty sysadmin all right so before I even buy was a shitty sysadmin I was just shitty helpdesk agent and this really where I learned a lot of the mistakes and the Phil pause that I've made in my career and hopefully you've seen these in the past and hopefully won't commit these again so this talk is unlike a lot of the ones you've seen previously these are kind of little visual cues that will help me spur my memory and then tell you some more stories okay so let's start talking about helpdesk and the where we started making our mistakes and that's in the password I remember this very clearly it was November 2004 George Bush had just defeated John Kerry in the election and the first password that we were supposed to give out to Orion users when we reset their password was close now remember this is the fall of 2004 so what would our password potentially be fall 2004 you see the problem is is that seasons change roughly around the same time and it is your 90-day password policy so I guarantee and that organization where I was working there still passwords out that are probably like summer 2019 so again if you're an attacker trying to spray an organization using these default passwords was the first thing I made as far as mistakes now you've probably seen something similar this in your social media you know find out your Star Wars Sith name or whatever by entering this celery of information sharing it with your peers you don't realize you're sharing all this secret password reset information potentially so just be aware of your online presence along that same lines if you know ed SCOTUS he literally wrote the book on pentesting he has a great story about disclosing sort of password information he did it was talking about a pen test where they just owned the hell out of this superintendent of a school district the way they positioned it was they wrote an interview as a schoolchild interviewing the superintendent as if they were writing a report so some of the questions were what were your parents names what was your first pet name things like that and this guy totally just spilled the beans so be careful of what you're exposing to the public in whatever you're doing this is a kind of on the same lines of password questions these these aren't really security issues but I had to share it with you where I worked at a long time ago we allowed our end users to create their own secret questions and some of them were just awesome so here's just one of them this one guy was really cute and he decided to play that he was hard of hearing so we had to make the person on the other end to repeat the question over and over again starting really quiet until the point where basically their addition yelling the question so what was a good question that would just really make a datacenter just die laughing well this was what it was who's your daddy so the person was like what I'm sorry I don't hear and so the person the helpdesk person was basically screaming because your daddy before the end of the day and the answer was me now this is another one this guy wasn't just real creep but I swear this is this really is the his question the question that he came up was what are you wearing right now and yeah but it gets even better and I swear I by the way if you haven't figured this out I'd love stupid just an animated ug animated gifs and memes okay so that's basically what this whole presentation is so I found a Jeff and it was the best one because it literally was his answer all right so the answer the question is is weren't you wearing right now that is totally inappropriate so that was actually as an answer so anyway but the point here is is that you need to be careful of the questions that you're disclosing if they're easily being able to disclose probably not good security questions all right so I'm gonna move forward to actual real confessions and things where I see security being prevalent here the first is the concept of TLS three-letter acronyms if y'all familiar with that right it's we're in security we it's all about TAS the first one I want to introduce is the topic of an SLA does anybody know what an SLA is service level agreement right this is the metric in which contracts are defined if you don't meet your la then you haven't done your job right well when I was in helpdesk we were measured by another TLA this one's called an f CR anybody know what f CR is first call resolution that's exactly right as a support desk person just picking up the phone I was expected to resolve every call that I came across to a certain threshold what percentage do you think average run-of-the-mill support desk person should maintain 25 good good number anybody else 40 you know when when I was doing this RFC our ratio was 90% we had to close 90 percent of our calls the first time they call so think about this from a social engineering perspective if I don't get the answer I want from the person on the end of the phone I had the longer I keep them on the line the less their Ser rate drops so you really the person supporting you really wants to get the call done as quickly as possible so if it doesn't work you just call back and the chances are getting what you want as a social engineer it's pretty good so again knowing that the end user this the support agent has to maintain this like high threshold of call frequency presents some security risk in lieu of operations all right so this is really where I started losing my taste for Humanity and progressed up the ranks of a helpdesk I went from helpdesk 1 to 2 to team lead and ultimately I became a sysadmin this is really where I think we learned a lot of our security mistakes I became that the master to prod really who controlled the keys to the kingdom right and this is where I was introduced to the three envelopes that have saved my career time and time again I'm gonna teach you this strategy over the course of today's talk and hopefully this will save your careers as well all right what you see in a lot of IT organizations is there's lots of players in them in the in the organization and we all have a common view of systems administrators yeah we're not too highly looked upon except for maybe ourselves right you'll see that you know that we're not highly looked upon because we have certain things that we value above all others and if you're not familiar with xkcd these guys make amazing comics and they're so true to life in this one right here we're talking about a sysadmin what do they give it what do they care about its uptime right it's all about uptime when it comes to systems administrators now we're all security professionals here I hope you all are familiar with it the CIA triad come on let's let's all say it together it's confidentiality integrity availability right when it comes to systems administration not security they operate on a whole different triad all right y'all familiar with the AAA triad nobody all about availability availability availability availability that's what they're concerned about and so my point here is is as systems administrators people think of us as you know we know all the answers we're the technical folks but in all honesty we're really just treading water we're trying to keep our head above the ground and just keep the systems running however the hell we do it right so when you saw my intro slide you saw my my hacker handle it's a Rainmaker and I'm not really that fond of it but I really believe that the best hacker handles are given not chosen and that one was actually bestowed upon me most people think the Rainmaker that's like money money dollar dollar whatever no it's more the traditional term the the Native American rain dance to save the day and the my VP at the time gave me the term and a Rainmaker and I never went away basically we to date I did a datacenter move and our whole network did not come over like it was supposed to production now it is across the board and somehow I got it up and working thus I got the term Rainmaker I have no idea what the hell I did to get it up it's running online but hey but that's why all the names stuck so one of the lessons learned today is when you're making changes on a production environment don't do multiple ones at the same time because you can't attribute one to actually correcting the issue make changes very scientifically ok all right so let's get started and start talking about some of the security mistakes that we made that I've seen and have done in the past if you look at my LinkedIn you can probably see a couple of the companies and we fixed these problems since then but for example I used to work for a very large chain of restaurants and we had to go our grand opening we were doing the whole ribbon cutting for the Chamber of Commerce and the problem was is there was only one wireless network that riderless network was reserved for the point-of-sale for our servers and for our corporate IT network unfortunately this the owner of the company one-a-day public facing Wi-Fi for our customers so when you only have budget for one what do you do you shouldn't expose it to the end-users right that's exactly what we did so I hate to say that but there's situations where we cut corners for the sake of operations we had to give it a open to the users thank god another buddy breached it before we actually corrected the problem but again it's cutting corners and that's really what's the the crux of the problem here which leads me to kind of some other things we were gonna talk about circumventing of controls all right you see that just because you have a control in place doesn't mean people are actually using it and this is really prevalent when we start talking about systems administrators because we're the ones installing those sorts of things we know exactly how to circumvent them and so again these are some of the mistakes that I've seen done and done some of these myself I mentioned earlier November of 2004 this was also the same time that World of Warcraft they you know the the vanilla version went live that was that was the big one right it is anybody gonna play the new vanilla version that's coming out or has it already come out August okay well I'm totally gonna be into that so look me up later but the problem is is back in 2004 I was working the night shift and well I needed to do my molding for more than the multicore 40 man raids so what do you do when you're working the night shift you bring your laptop into work yeah that's the sort of the circumventing of controls either that were talking about what we've seen in or common terms now we've done you know Hardware filtering is is rather than bringing hardware and data into organizations we're seeing users take data out quite often our security controls are so harsh that it's preventing our end users to do their job and so they're taking their data and they're maybe chained only get out through Skype or slacks so people can they can access their files and their personal devices circumvent any sort of caz B's and things like that so just be aware that you know maybe these controls are maybe too powerful and that's introducing even more risk this one here this one was a fun one company decided to get wise and install content monitoring so you know seeing what our employees are doing they also allowed outbound SSH so a simple sysadmin who doesn't want the company to know what they're doing can create and outbound socks proxy to their AWS instance or raspberry pi at home and completely circumvent the filtering from your company these are the sort of things that your sis admissions are doing because we're giving them admin access to do you know whatever on their own personal on their their assets on their computer so again probably the people that are installing these controls are also the same ones that are circumventing them another one this one was pretty funny I hate my keys I am the most forgetful person I tell people that I have the short-term memory of a goldfish and it's probably true I forget my keys with my key fob all the time and so what we would end up doing is because of network segmentation we would allow for multi-factor to access our PCI environment at the time rather than having to use my key fob to give it and I happen to also be the VMware admin so we just jump on vCenter get console access completely circumventing the MFA those are some of the controls that we got to talk about and that's what we have to do is watch and monitor our systems admins because they're gints are committing the controls we put in another one here is we're lazy that's how why we got into technology so all machines can do the work for us many a time have I set a job off and walked away with the computer unlocked that that happens more than then I would like to admit in the past so what we ended up doing was we created a distribution list called donut and I love this and this is such a simple thing because it game if I as the process if you happen to find a machine that's unlocked and you're able to send an email from that user to this distribution list they're obligated to bring Donuts in the next day yes if you've ever wanted to see a grown man clear three cubes in like rapid succession to get to his unlocked computer that's how to do it so what the also here you can see from our lazy sysadmin is you know the machines unlocked this is something that again I now that I'm I've learned and you know recovered from my mistakes I used to let vendors come into my network unencumbered we might monitor them for a little bit you know set up a go to my PC session so they could get into the systems but was the sysadmin actually watching him now it's probably having a smoke break or talking to Sally in accounting the point being is is I wasn't watching our vendors and our third parties accessing our systems and just exposing our organization to who knows what same thing is applicable with us well they target their breach came from an external vendor so be exposing our vendors to our network unsecured Lea is not entirely the best also this one you see that the forest and the tree here right this kind of teaches me our Tala tells me about responding to alerts one of my jobs previously was a tripwire responder and responding to the alerts that came in and there were just so many so what did we do because we had just so many alerts we just ignored the hallo that the problem is you can't see the forest through the trees we were tripping over ourselves so pay attention to the alerts tune them down because otherwise you're just getting a bunch of crap and there's no value in it so - and you're tuned your logs and make sure let's move forward I think this video will play I don't know if the audio will but you can probably get the gist of this anybody watch Saturday Night Live anybody know who this is Nick burns your company's computer guy so in the process of being assist admin not only was i jegg a gigantic asshole to my customers like Nick you'll notice that Knicks logging on to the machine on behalf of the user remember mmm yeah what he's doing is and the circumstance is you know putting in his own password potentially and so now that I've learned about this what he should have done is after you know messing with Jennifer's computer here rebooting making sure that you know Nick's MIT credentials aren't stored in memory so the point here is is be nice to end-users and if you're going to log onto a machine make sure that again you're the ntlm to memory or I'm sorry ntlm V to hash is stored in memory so hypothetically saying Jennifer's machine got like a keylogger or a ram scraper if I go over her machine login to a machine well now my creds are in her memory exposing it so be aware of that all right moving forward passwords you see this is a password recycling here and then an island of trash in the background because that's what we're doing here when we recycle passwords it's trash there's a couple stories around this one let's start here I have an organization that I consult with and they use the same password across multiple different platforms what does that mean exactly well it's an ad account it's a very very highly privileged account but it's running an application running several service accounts it's also running hard coded in several different automated scripts it's also running in their Linux environment as well facilitating some ad bridging in some processes the point being is is this one a privileged account is bound to like six different applications and processes so let's say we Institute a password policy where that password gets rotated you know what happens nothing there gets an exemption in your audit policy and that password has never changed what really needs to happen is we need to make our credential single-use making sure that they're located and used for a single purpose and a single purpose alone another story I want to tell here - anybody know what this movie says kangaroo jack it's a terrible movie but this guy he was the sysadmin that ran our UNIX team and this was his icon on our I am client so this reminds me of what he did so compliance mandates in certain organizations require you to rotate your passwords and so this guy got smart he got really smart he decided to send a batch job that changed the password across the enterprise so man main maintaining the compliance but what he didn't tell they come auditor was he just ran that exact same batch job again and reset the password right back to the original so the says the password change right on the logs auditors happy right but did the password actually change now did security change oh the point being is quit reusing your passwords it's really really a bad thing which leads me to a really important point about compliance compliance is a low watermark when it comes to security it is the minimum viable product across the board however this is also probably the biggest carrot on a stick for organizations if you haven't followed hermit hacker this is a personal friend of mine and a really good guy and he told me that using compliance as that carrot on a stick as far as motivating businesses to spend money on technical and computer security controls is how to go about doing it so tying compliance to your your security controls goes a very very long way so I've shared with you a couple of the mistakes that I have made I've definitely made a few I'm gonna share with you some mistakes that other people have made okay we'll start outside of the worlds of systems administration see if you can pick up on this one anybody know this this one all right let's let's watch Kanye over here you don't need audio for this one all right so take a look at what he's doing he's showing President Trump the new explain one I believe oh you already saw it all right hold on so he's showing the president right here he he calls it a gift I'm still stuck on Jeff it's Jeff for me I show of hands is it Jeff gif oh did you suck now but take a look zero zero zero zero is his password and so that really kind of shows me about default passwords yes alright so this is a real confession I actually had a system that I maintained and I was the junior admin and I was told to change the password and my senior admin said no we can't change the password we have to keep it at the default I was like why it's like so the vendor can get in when they need to okay what the hell are you talking about like much like dr. Phil here it's just like one of those brain farts so when it comes to passwords I need to make a very good clear point guilfoyle says it better than me if you're not familiar with Silicon Valley I highly highly recommend this show it's more like what does they say it's not like social engineering it's more like natural selection right this is not hacking here don't post your passwords on notepad sticky notes things like that here's just a couple of the ones that I found over the course of just visiting clients and even more important if you're gonna be on TV holy cow do not put passwords in your workplace all right worse yet just if you're gonna be on TV don't have it done on your work site be away from around your machines maintain that clean desk policy but yeah these are all just people that have had passwords just exposed because they left them on TV crazy stuff right alright so let's talk about Miss configuration if you think about it there's more security incidents due to Miss configuration than there are security vulnerabilities there's way more people installing applications than actually developing them and so what you're seeing here is Gartner says are really good that 99 percent of cloud security failures will be the customers fault by 2023 it's not the s3 that's that's compromised it's the configuration settings so we have to make sure that we're doing our deployments we're doing our work diligently otherwise we're exposing our applications to undue risk all right so again it's their configuration that's really the the scary part alright now who here is familiar with truffle log this is a really cool tool that I want to show a problem that many developers make on a daily basis and it's a hard coding your credentials into scripts this is a just freely available tool up on a github and what it does is it recursively searches through the commit histories of github submissions what that does is you know if you put your password in clear-text by accident you make a change you submit it if you don't remove it from your revision history it's still just as good there so I want to show you a couple things before I begin and show you this it's a true - case here all right so this is github just from this morning you can actually just search for keywords and it will show you the most recently indexed accounts and you can see these are people's secret keys just right out here and get up huh yes oh sorry I'm not showing my screen thank you let me a duplicate my screen here do you do all right let's see here yep so here's the github right here you can see that this is just indexed recently and people's secret keys are just exposed in clear text here and if you want to automate the finding of that this is really where truffle hog is very helpful so I created a script to make this a little easier today let me see your si desktop truffle hog - and I wanted to show you this idiot that put all this open light his SSH keys passwords all of this stuff is in this morons github repo I mean take a look at this all right I'll scroll up here so you can see a couple SSH keys private keys come on scroll up damn it there we go so yeah there's a secret there this is all in one single github repo don't worry these this is my github repo so it's they're not actual real passwords so yeah tough yeah it's not you're not finding real passwords so but my point is is that you know developers easily put passwords in their scripts their source code and it's very easy for an attacker to find it all right so I'm gonna switch back into presenter mode and here let's see all right Oh No that's the last time I'm leaving the screen here so we're good there all right so there's all right all right so let's keep moving forward we got mr. t here talking about April Fool's this this is a real story that happened in one of my organizations well we had a web developer that was let go and he was not too happy about it so as one of the last things he did before leaving the organization was a planet a time bomb on the e-commerce site so at some predetermined time when he was long gone lives like nine months after he loved the time mom executed it just happened to coincide with April Fool's so what happened was is it was actually fairly benign logic bomb it basically was a JavaScript that inverted the screen so it looked like you were looking at everything upside down happened like I said on April Fool's so what happened was is the company played it off like it was an April Fool's joke but in reality InfoSec was running around with their heads cut off freaking out so I thought that was an interesting one nothing really came about it but again this just shows where you know potential risks exist in your in your processes this one here this was a good one I had a client of mine that we were doing an assessment of their privileged accounts what cyber-ark does we work on privileged accounts security so your most important passwords like domain admin accounts need to be managed really aggressively and we found this domain admin account it was a systems admin account that was running a service and it wasn't just a regular domain admin account it was the guy in the room that was well I was given the presentation too and let me real quote him word for word I said I think he said I'm not calling you a liar I just don't believe a word you just fucking said yeah that really makes you scared right so I did what any rational human being would do and when I showed him that fact that he was using his own account to run this service I handed him the HDMI controller and said let's log on to the box and take a look oh it was amazing within like ten seconds this guy turns bright red redder than the guy shirt in the back and it was awesome he said I'm so sorry but the point is is this guy was running you know service accounts with his own admin account in the event that these service accounts are running that meant that those credentials are stored in memory so in the event that that machine was compromised an attacker would have access to his domain admin account pretty scary stuff right now this is one that I shouldn't have to tell you but I'm gonna do it anyway you shouldn't be surfing inappropriate content at work right this goes this is a great story because I get I get to shut somebody down hard so on my one time I was managing our sand storage and we were about to put a my Gration so we're cleaning up data and I found a treasure trove of inappropriate content what happened was is one of the people on our team had a really weird fetish but I digress he happened to be storing all of it in his My Documents folder on his corporate issued laptop what he also didn't know was that in My Documents folder was being replicated back to our standard for document retention purposes so what he was into was getting synchronized back to the files here we happened to terminate that a couple years ago but the data was still there and it was still being backed up on a regular basis which is how I discovered it so this was the cool part so he accused me of planting this you know contraband on in his file share so be that he actually started before me I was like okay bro let's do this pick a day any day pick one before I even started and let's restore that file so he's no longer there and I was so yeah but my point being is is it I'm not trying to can't shame anybody just keep that stuff off your per your work machine do that in your own time right all right so I'm gonna kind of segue into some things that we want to talk about words that you shouldn't be using in a professional context these are things that I've seen people use some of these are not a little mean they're not bad but again I would advise you to try to use different words ping ping in the context of Technology is a beacon waiting for a response in the terms of business it doesn't necessarily mean that it's morphing you say if you pink somebody in accounting it's a lot different than pinging somebody in networking right they don't quite get it all right another thing is is the term to be honest I hate this term because this implies at times you may not be honest so just don't use this word you know just again try not to use that another one the the concept of burning the ships that you know that's the hell or high water we there's no going back you guys familiar with where this term originated from the conquistador days of like seventeen sixteen hundred's Coronado made it to the new world and burned his ships so that the the his soldiers had no recourse but to discover the new land you know what that never actually happened there was no burning of ships rather than burning of the ships Coronado actually scuttled the ships he programmatically broke down the timber and the linen and all the resources of the boats that's just stupid so do things smart okay another term that I want to talk about today is the term brand jacking this is some marketing mumbo jumbo it's the same things that we've heard of time and time again it's fishing right let's call a spade a spade and not use divergent terms because it all means the same thing this is fishing another term that really really gets me this grinds my gears is the term open the kimono anybody heard this one I don't like this one this one you we see c-suite executives just within the last couple of years using this term this is a reference to the historical times of Japan where the geishas would you know open their kimonos as a show of exposing their naked bodies that's not the intention of today's c-suites using this term it's more of being transparent honest and open with your transactions without knowing the historical connotations this really changes the context of this phrase not only is this sexist but it's racist and so again in my opinion just best not use these sorts of terms in the context of a business discussion right another one that I really want to talk on and I think this is a really good opportunity to talk about this is using the term hacking in relative to criminal actions you know what there's a lot of hackers in this room today and none of you well I hope none of you are legitimate criminals hacking is not a crime you know this is how we're finding these flaws and along with the whole concept of the hacker and the hoodie and the with the face mask these aren't hackers then you want to talk about what real hackers look like these are real hackers well they may not be this is a this is not a real person calm these are just computer-generated people these are the criminals that we need to be protecting not against these sorts of folks you know so you've kind of shared with you a bunch of the terrible things that I've seen in my past as a systems administrator I want to share with you a couple of things that are actually good things that you can take away one was just using common sense being able to really protect an organization from ransomware somebody in accounting really shouldn't be running you know bash shell scripts or you know VB ship scripts so disassociating that type of extension to something like WordPad and so is just a really easy way to go about doing that and yes that is a base64 Easter Egg there so another thing that you can do is changing the default installation path just going from C to D what does that do from a security perspective a lot of the script kiddie tools are looking for default installation path default registry values things like that and so by just deviating even something as small as that you can actually break some of these automated scripts now let's be honest if we got an apt in the house and they're actually doing byte by byte you know file by file scans this gonna help but if we're talking about some script kiddie stuff yes this will help also from a systems administration perspective this will actually by breaking out your OS volumes and your data volumes will actually shrink your data backups too so it's a win-win another thing is is that the personal relationships in being in a career of a systems administrator you can see these are some guys I used to work with at fossil that are now we've gone our separate ways but you know I still run across these guys as clients I'm not gonna tell you where but this is a government facility I mean that's crazy right there's a these are my two best friends we used to work together between the three of us we have 17 girls yeah I know there's like no y chromosomes that were produced at all but these are some of my best friends that I wouldn't have had if I didn't have a career as a systems administrator and then we wouldn't have the awkward hugs from Jason - so by the way if you haven't got a chance Jason's do want to talk later tomorrow I believe and it's gonna rock your socks off so I want to give credit to him and also if you can see we can have fun on the job - anybody seen American Psycho yes so you get that reference right there feed me a stray cat yeah so I changed all the payroll systems to say feed me a straight cat for a day just because I wanted to so I've kind of talked about you know being a shitty Systems Administrator I really do believe that the shittiest of systems administrator can also become one of the best security professionals because again we've made these mistakes we know how the corners have been cut we know where the skeletons are buried and so this kind of allows me to segue into how to get into info second how to get into security I get a question a week online from people and so one of the questions I really get a lot is is you know the concept of should you go for experience or should you go for a certification I kind of believe in both I mentioned that you know I'm an adamant fan of is c-squared a conte ia and all the different certifications there's a method to it anybody know who this guy is here Dave Ramsey I love Dave Dave got me out of 60 grand in debt it's all based on this concept of the debt snowball and taking the small bed debts and rolling them into the next one and once you get that paid off you go to the next one I highly encourage you if you're in any sort of fiscal debt take a look at Dave's course it's worth its weight in gold but if you apply it to the concepts of certifications in IT it just makes it even more sense so this is how I started I started with an A+ security we want the security plus and these build off of each other so again the harder that gets the you've already kind of maintained the baseline from the previous certification so I went from the SSCP to the CISSP and then there's all these different other ones that you can get now this is just a handful of governing bodies there's so many other different ones but you can do these are the ones that really go into the the crusader defensive style now if you want to go into more of a Red Team offensive certification track there's also a different certification track you can do as well these are just a handful but if you really want to hone your dark wizard skills definitely go down the route of a couple of these certifications OSC P and OSCE in particular really good ones but there's a problem with certifications they're only as good as the the paper that they're printed on so if you're going to get any of these certifications know what the hell you're talking about and demonstrate that okay so in lieu of doing a QA today I wanted to kind of make this an opportunity to kind of have you share with me and the rest of the people today a couple of your confessions a couple of these stories of mishaps that you may have done in your career and lessons learned from that so if you don't mind if you've got to kind of share with the share with the class I think show anybody want to just show you AG share with us serve right here up in the front tell me of one of your confessions here if you if we have to if nobody else volunteers then yes you do [Music] password zero one very complex okay oh no way so you're saying this rogue accounts started jacking in with accounts no oh no so let me get this straight you kick off this automated script that's supposed to disable a few accounts what ends up happening when you run the script and what did it really do [Music] so you shut down everything in the production domain by accident from running the script yeah we need yeah thank you and question are you still at the time were you still employed after this event Wow that's even more impressive right on right on for the sake of the time I'm gonna uh show of hands anybody other stories real quick you'll want to share with the class one more back here yes sir yes on that same note don't take WebEx calls in the men's room I purposely flush multiple times just to spite you when you do that okay thank you guys for your for your confessions if you may I want to share with you a couple others that were submitted to me online before we kind of conclude today's talk this one's a pretty good one anybody know what you name s does on a Linux system anybody it simply just checks the machine name properties of the remote system right okay very simple command that says Evans do day to day all the time right anybody know what that same command does on a Solaris system now rename the host yeah I can do with some damage I mean moving forward how many of you all have done this alright so show everybody's done then these are mistakes we all make right very quick easy fix to this simply and I've started doing this to my junior admins I put this in their profile and it has fixed so many unnecessary deletions in your organization so again just making simple fixes to correct that I saw somebody raise their hand earlier in the back did you all have one more story you wanted to say no yes Wow be careful of the commands you run very scary scary story there I got a couple other ones this one was given to one of my buddies and a DHA they used to run a they did remote support for restaurants and they had their credit card processing available via dial-up modem and so this ingenious young professional decided again this is a long time ago maintain a password list and decided to put it up on Geocities because nobody would ever go to this unlisted website right well funny enough there's this thing called wayback machine that password list which has never changed still works I swear to God so I mean this is some scary stuff we're talking about another really really good story came to me from my friends at fuzzy snuggly duck if you're not familiar with them look them up they're awesome incident folks this is a really good story about how to properly handle an incident in this circumstance we have a database table that was compromised by an attacker attacker got into the network who was able to take X fill this data but not just X fill the data the attacker decided to be smart and basically planted their own canary back in and replaced one of the records in the database table so eventually the attackers were found in both sexes finds the the event and alerts the end users hey urines your data was compromised and we are we're on top of it right so the problem is that also ended up notifying the attacker because he caught her up did the data oh it gets better than that so again so InfoSec has now sent an email and they're you know company letterhead proper formatting you know telling them that hey you been breached so what's an attacker gonna do use that same letterhead just spam the hell out of just the same people that he just pwned so what the attacker ended up doing was said hey yeah you're right we have been compromised in order for payroll to be successfully done you're going to need to log into your this phishing site and give us your admin passwords so not only did they steal the data they also fished the attacker afterwards so all right so I've got a couple of things before I conclude and so again to make sure that you actually respond to me I'm going to bribe you none of this like small stuff I got a real full-size candy here so I told you there was going to be a quiz later all right SLA all right I can't hear y'all so I'm just gonna throw stuff okay by the way I'm so thankful there's no projectors like up here because if you've never seen a guy had a 4k projector with the like a Hershey bar it's not pretty so personal experience confession there so here we go all right service-level germane FCR yes good job CIA all right AAA all right let's see if I'm get a spiral on this one I just wanted to put this in there so all right so again wrapping up guys just a couple of things too before I finish you know prioritize your security over operations that's it's an important thing don't cut corners it's kind of important the fact is is that the systems administrators these are the people that we're trusting to install these solutions they're probably the people that are circumventing them so let's watch our Watchers right be aware that you know just because the application may be secure maybe the installation itself isn't so be aware that misconfigurations are you know in jeopardy as well and the fact of the matter is is vs. admins we're humans too we're lazy we make mistakes there's errors that happen risk is everywhere learn from the mistakes that you've learned from today from myself the people here as well and the people that shared with online the confessions so again hopefully you won't make the same mistakes this kind of leads to me back to the three envelopes story that I told you at the very beginning when I first started at my very first sysadmin job I got three envelopes in my my desk drawer from the previous systems administrator and said hey this is what you need if something really goes awry and you just don't know what to do open one of these envelopes and it will really help you out so the first one when the production system went down I didn't know what to do I was about to lose my job so I decided to finally get that envelope I open it up and blame the previous assignment cycle that's all Sean's fault you know what it actually worked it actually kept me back I got my job at all good so about nine months later same thing happens again production goes down again things are going crazy I have to go back to the drawer I pull out the envelope I open it says blame the vendor so that's what I did work like a charm about six months later the same thing happened again so again going back to my trusty envelope I open it up it says prepare three envelopes yes so that concludes my talk thank you so much everyone I appreciate it

Channel: Adrian Crenshaw

Views: 1,374

Rating: 4.5 out of 5

Keywords: irongeek, security, hacking, infosec, saint, louis, showmecon

Id: ifs_hIlf_3o

Length: 50min 14sec (3014 seconds)

Published: Thu Jun 13 2019