VIM tutorial: linux terminal tools for bug bounty pentest and redteams with @tomnomnom

I'm in London together with Tom nomnom and we just finished a great hacking event it's time for some educational concepts absolutely like we just been hacking on goober yes so I thought what we might do is like just show how I might approach doing a little bit of recon on the target huh with bash and with Linux and all that kind of thing cool so I've got myself an empty directory yeah like the first thing you want to do is like some kind of subdomains enumeration and there's like a whole bunch of tools around I'm just gonna use one that I hacked together awhile ago that I called asset finder so my github repos wait and you built this one yeah it's super simple it's just like fetches from passive resources didn't do any actual brute forcing okay I'm just gonna use it because it's quite fast cool so like let's have a look at ubirr right yeah so if I run this I'm gonna get like fairly quickly a like a reasonably big list and I want to store this in a file and I'm gonna do this with this angle bracket character here yeah so basically what that means is I want to take the output of that command and put it into a file yeah and I'm gonna put it in a file called domains so if I do an LS now you can see I've got a phone call domains wait okay now not all of these are gonna have like HTTP servers running on them no so I want to like do a pre-filter and go and find which one's of those do so I'm gonna take my domains I'm going to use this character here the pipe yep and what that means is to like take the output of this command and put it into the input of the next command so I've another tool called HTTP probe okay which takes domains on its input and checks it furs and HTTP or an HTTP server listening and if there is it outputs so you're taking all the all the domains that all the lines that you got in this domains file and you send them over the edge exactly man could you do the same thing with what weather yeah probably I don't really use what web but I think probably and if not there's ways around it you can you can do anything in bash man so I want to store the output list in the file but I kind of want to see the output as it's happening as well I'm going on it so I could use that angle bracket thing I've used earlier with you another file or I can also use the T commands which like T like a split right it takes one stream of input and splits into two so one half goes to a file and the other half goes to your screen hmm so I gotta make this in a file called hosts and we can see the output happening now so these are all the HTTP and HTTPS services bound actually send me requests out now yeah yeah for record hate the implement we see a big time sorry exactly that it doesn't care what answers did you see eyes open or not yeah is it listening yeah and if it is a print it in the screen and because we piped this into T it's gonna get stored in the file - mmm which is awesome because that means I can like reuse it for stuff it's perfect I never used T I always go back and have to look at my file afterwards yeah well I can I do and I do the double one in four words and I'm continuously writing to the same file and that kind of stuff yeah I've done that a lot so you can do T - a to append just like the double arrow yeah yeah okay for a Penta yeah so I really like that because sometimes I have this thing where I'll do like the angle bracket yeah now leave the command running yeah I come back an hour later and like it wasn't running the whole time like it was just hung yeah but nothing's been written to the file no but with T like I get to see the yeah absolutely like I get to see his drawing past yeah like nice and fast can have a look at that file you can see we've shot 341 hosts to look your whims of laughs just be fast literally there's no shortcut it was just a queue or what I want to do now is go and like I know that this is HTTP servers listening right yeah I want to see like what's there and just have like a poke around I'm not looking for anything specific I just want to know like what's running on these systems you know I'm gonna use it took on Meg okay it's another one of mine I have a lot of like tools of my github profile I built for this kind of thing and it's pretty good going and fetching one path like lots of hosts okay so if we look at this how proud but you look quite a bit here but we can see you got lots of options like well your concurrency level is so it can do lots of requests concurrently yeah if you've got multiple CPUs that will be in parallel and one of the really important things here is you can tune the delay between requests are the same class so sometimes I might say like I only wanna hear each host once every minute something like that so you get control over that you can set a bunch of other things as well so let's do that so I just look Felix watch which is just the root directory and I won the delay to be what did we say one second or something like that let's go for both modes as well because here yes we can see what's going on unlike off it goes it's gonna run a whole lot of stuff it's making a request to all of these and it's saving us out but we've got the concurrency at the default levels which is like 20 it can run way faster if I like to do really really fast yeah it's definitely not the fastest out there so James careful I'll be nuts as an amazing tool as part but called turbo intruder yeah if you're looking at lots of things on one host it's so fast yeah so but this is kind of almost deliberately slow in a lot of ways but still it kind of works out okay yeah what do we get and so I've got a star actually called output cool huh so I old CD change directory into that huh and here is one folder for each of the hosts that we looked at okay and if you become say like this oh hello kuba ka's yelling there's two files in there okay and each one of them is one of them will be the HDTV and want to be the HTTP okay there's an index file in here that includes like all of the file names of all and the most sand things and yet response code was cool but you can easily grip that this to find which door 200 or which are you're doing you and all that stuff so you can know map your attacks of use even more yeah for sure so when we've been hacking on uber right we've been we know that they have stuff this on uber internal dot-com yeah we know that right from their program page is Cisco so I might want to look for uber internal and all of these files like if I could type that would be super useful so I could use these flags all the time without really thinking about it the H isn't actually even really needed but that means display the file name yeah and is displayed the line number ah is recursive so it goes into all of these directories and I use case insensitive so okay shows up with like a capital u is still gonna so we got quite a few results out of this and they all look kind of similar and they're spread across lots of lines yeah so because like this isn't the best environment to really review this output only things I tend to do is our pipe things the pipe into VIN yep so Vince based - tells them I want you to take the input you've been given from the pipe then open that in so now I can kind of see all of the different places that uber internal come as shown up and I know I've got 34 results over like a buffer yeah and then you're able to play around with it and if you want to save it you either you save it somewhere and you can continue so you're right I can just say this is a park of results : W and results yeah so so in VIN I can just hit /yeah say like take me to the uber internal path since on ROI and highlight them for you and I could hit and and go through here but one of the real common things I face when looking at this kind of stuff is there's a lot of stuff in here that's clue like it's pretty boring I don't care about it know there's a lot of this content security policy stuff but one of the nice things I can do with them is I can take the current file which is represent by person and a bang which means it runs through a shell command it's like typing this content this data into a process and then back into their no way so firstly I can sort this and then the other thing I can do is say well actually I don't want all of these content security things alright yeah so yeah you just curate being the output right now yeah absolutely so that's nice so we can have like 10,000 lines you like I don't want to look at all these manually yep like that's a real nice way to work because sometimes you have to find exactly the thing you want yeah but it's pretty easy to find lots of things and then pull stuff out that you've done yeah and it's super fast too because like the stuff that's in them might have taken a long time to search through yeah this is like a gigabyte of stuff yeah there but like once it's in here I can reprocess it really quickly so we have like these CSP things as well like they don't want those so I'm gonna do the same thing really VI for servi is invert the V invert yeah which means like remove the eye casing so TIFF and I gotta pull out the CSP ones as well so you watch him go there they are so I'll often go graphing for lots of different things so I have surprise surprise another tool called gf which is short for grep for and what this does is it takes files which are stored in your home directory India JSON files an Aikens pain patterns these are patterns that are plastic wrap and the media this whole tool is just a wrapper I can't remember this whole thing I I can't take this out every time I want to use it so I have it stored in here and I can just run gyah AWS keys and if there was any here it would find them yes I have a whole bunch of those for finding things like s3 buckets for example or there's no I'm here a spice thing it is here so base64 right so everything inside information that you got down you saying gf and you're pretty fine like oh it's like look like reg X to me yeah this is it absolutely that's what is this is a regular expression foreground yeah so the form of these files is we give it the flags that we would have given to graph yeah like we can run this exact thing and get exactly the same output just by like copy and pasting it so like the outputs the same but wait used to type this right you see ey j that's the basic C 4 encoded JSON string like the start off so if we make one like this pass it into 64 like we see the output starts at UI J yeah because that's the curly brace and the quote basically for encoded it always looked the same yeah and I have a bunch of these which prefixes for things like serialize pH P values serialized Java values XML this one is these are HTTP at HTTP the URLs sometimes you see base64 encoded URLs yep and a few of us so we've got these base64 encoded things and like I don't know what these are right and I can't read base64 so right let's do kind of the same trick we did before yeah the min of them and I want just the base64 encoded fast because I want to do something with them yeah so I'm gonna use my percent bang trick again and I got a person you ever think of all yep which is a text processing command really it's a whole language okay and I'm gonna tell that I want my field separator to be the column and I want to print the third field same thing as cut - Dean yeah you can do it with cut - for whatever reason walk handles different kinds of whitespace a little better than cut does so sometimes you use cut a new Finder doesn't work because you said you know separators a space yeah and actually there's a tab and it just didn't work or kind of handles that situation a little bit better so if we run this we should get just that base64 encoded things so now we need to remove some characters in the beginning yeah so when I have stuff like this in vim like I could just like manually delete all of them but fake ctrl V that possibly a visual block mode yeah and I can do vertical selections - are nice and then I X and they get to each other so how would you do it if you ended up with getting a Leslie we want to remove all the nines here on the end believe you it will end up with a dot on the end when you get a DNS file okay should we put some dots on there yeah okay so I've got a couple of options here so like the sort of easy but slightly more tedious way is I would hit shift a which is append yeah an impact space and then I couldn't escape that's yeah one line but a nice thing about vim needs to be picked off it repeats the last thing you did yeah so if I hit down and then dot dot dot scan and then I can just alternate between those two and then they're gone right so that's one option the other option is I can search for something like a dot that's at the end of the line so a dollar means end of the line okay in them you can do a search and replace and if you don't put in the search part which would be here this would be the so yeah that's right if you don't put one of those in it uses whatever the last thing you searched for words and because I want to replace it with nothing I just got another slash in and then they're all gone to do right cool so we've got a few of these basically foreign country strings we want to know what they are so but there's some duplicates right yeah so I'm gonna pass them into sort - you with the percent bang trick and then not only they're sorted but the duplicates are gone yeah don't care about those and now we want to know what they are so we kind of know that we could take one of these things and if I just run SH in the gone thing it'll put me back in a shell temporarily okay so now you put down on hold you've been still running in the background and when I exit this shell again it'll come back up yeah so I know that if I echo this into base64 dash D it will give me the decoded version yeah and I want to do that for all of these lines and here we only have four but you know you might have 400 yeah and you don't want to do that manually no so percent back again it's coming back and we're gonna pass all of these into a tool called backsides okay no X I never really understand what it does okay so excise takes multiple lines of input do you give it with a pipe example and it will run a command for every line oven and it's a little bit tunable there's a lot of options to it by default it would give all four of these to just one command or one go you don't want them so we do n one and one at a time yeah one at a time I don't know where I stand or like what is short for but what this means here is I'm gonna use this racket sequence of market closed bracket it's like a placeholder to mean that line of data and I'm just telling X arts that's the pattern I'm gonna use I'm gonna run the shell the sh-show with a command echo whatever that line is into base64 dusty right just like we did on the command line a second ago yeah and because we're doing a percent bang that's gonna come straight back into our little buffer ready there it is so no these are the 64 decoded versions we can see that these are all like fairly boring in this case but have one nice example of what I can do with all of these files because there's still a lot of them but I spent a long time with a really slow internet connection oh so I kind of got used to doing everything on a VPS yeah in I use digital ocean and I want to do everything your SSH because desktop on the next no I I want to do it in the terminal on my nice fast speed yes so I do have like an alternative to screenshotting which is kind of is the low-tech version that's being triggered so what do I do is say like I've got all of these files uh-huh and so I'm using find to find them all and saying give me just things that work for and I have another tool called HTML tool and watch team a new tool does this you give it a list of files and then you can have it pass things out of the HTML so if I was looking for all of the JavaScript files yeah for example if I just have a look at the help but oh there isn't one I see no ribs something like that yeah there we go so this is all of the values of all of the source attributes in all of those files and this is all of the like so some of them it's gonna be images some of its gonna be JavaScript files because like the image tag has an SRC attribute so does the script tag okay and wait we can fix that pretty easy and just say I want just things that end in dais with crap and like now we've got just these are all the JavaScript files HTML files and see do any of these look interesting and I can do the same thing I did with them before and put them in there and refill to them and change them and stuff cool but the other thing HTML tool can do is give you the contents of tags so here I'm going to look at all of the title tags and I'm going to put these into a bimbo fire so there's a lot that are gonna be the same 301 moved permanently and what kind of thing so that sort so - you for unique here we go so these are the 23 unique titles that were in those one hundred and something or two hundred and seventy nine different ones which is amazing but how do you know it's fight would use where this uber self driving is that's a good question so one of the things I can do here is I have a VIN shortcut set up for this which I have set to leader G which does a graph for whatever is under the cursor and then opens the results in a new buffer in a new tab so it shows up in a few places by the looks of things but this is how throwing kuba calm looks to be the so I can hit ctrl W then G shifter and it's gonna show me it's take me straight to the line blind 25 where that uber self-driving thing shows that I could just add a line graph here we can kind of see all of the rest of the shows up no wonder you were able to get through their own github repo in seconds all github repos then you stumble upon so for the game we both like yeah something's real similar this thing right yes so I tweeted this a couple of weeks back and then like a few hours later I used it you have to find something pretty bad that we like this is a mess right it's all on the line it's not easy to figure out what this thing does so we can go and like see what this does yes as an example and then maybe we can like pull it apart and explain all the paths so cool I'm gonna hop into one of my repositories it's not a security late thing it's just another tool biro I know I've said that a lot this one is cool thanks man so this isn't a security tool this is it took off from and what this one does is take a JSON file or some JSON data like this mm-hmm and turns it into a series of JavaScript assignments okay so the reason for that is that I can then grab the stuff that's the G and Grahm is is graph it's like a boatman hot Mentos Bob hunter of grep in JSON so I can take like just the likes wines from this and I can see the full pasture right yeah yeah and then I couldn't do the reverse to and like turn that back into JSON again we're just that okay so this is the repo we're gonna use so this is the thing that I kind of tweeted yeah and what it does is it takes all of the get object files and history and everything has ever been in a git repo and outputs it all as one text string so that you can run it for ground and so for example I might want to find all of the times that I made a go Punisher or something like that and this probably gonna be a lot of results here here we go here they are like every single one that's ever happened in this repo yeah so when you have a git repo there's this docket folder yeah in there is a follicle objects and those objects are every version of every file that that ever has been yeah so a commit itself is actually I think is one of those objects right yeah and it has a pointer to a thing called the tree which is another object which has appointed to other things like blobs and other trees which are just objects again right so like my license file here it is just another object with my license of it which is apparently from 2016 okay who said of you is going for the file name you're going for for the identifier yeah from that time yeah three so note that like when I looked at this command they point to a particular tree yeah that tree describes the state of my repository the time okay so we can travel back in time yeah so the object file stays that's how you can like go back in time so if I to get checkout master and then this three currents means three commits before master it's going to take me back in time and the reason it does that so fast is the files never went anywhere they're still in that objects bar actually yeah I can go back to must over there - so let's have a look at that command and and will reformat it and like make it a bit more readable and hopefully we can make some sense of it so I'm gonna call this something like gah don't get speed um dot SH yep and the first thing we need here is a thing called a shebang which is a hash and a bang and slash bin slash bash and that's actually an indicator to the colonel that this is a shell file that needs to be interpreted by the Baksh interpreter we're not having you there able not execute right yeah absolutely so I'm gonna take the graph off this because it's we don't kind of don't need it and we end up with this great big long thing so let's add some line breaks in here to kind of make things a little bit easier to read yeah okay so the first thing we have here is our find in docket get object pack for file names that end with dot idx yeah so after a while you end up with a lot of object files and get and like too many right so as an optimization yeah get will pack all of those together into one file called a pack file like all squished together yeah so it gets better compression you need to know where they are in that pack file so it has these dot idx files that explain like stores like what the offset is in that file so we find all of the index files and we put into a while read I and what that does is take each index file name and put in a variable called yeah and then we really command with it we say I want to do this command get show index and pass it the I there either and this left-hand angle bracket means take the file on the right and attach it to the input of this command maybe five to the right or to the left sorry I get a little door this last done here yeah takes the file on the right and like attaches it to the input of this get show index command so that show index command outputs three columns of data and we want the second one so we use off to do that and then we're kind of done for that that gives us all of the object hashes that have been okay then we want to find all the objects that haven't been found so we do a find the process now yeah absolutely that's exactly what we're pulling out the park files because we've already dealt with those yeah so we might as we yeah means we are we are ignoring those absolutely we are using orc to but no got a capital F here in north yeah so this is the field separator I am so let me pull up another terminal earthquake so if I run find doc Gair objects this is the the output that I get and we've got type F as well which is just files so the moment we see directories now is just files and this is the bit that we want the first two characters have been split and somewhere in here there are some pack files too here we go there's one right here we don't want that one so that's the grep - be a flashback do not want that well what we do want is this with the slash removed this is the last two parts of this path if you separate them by a slash yeah so we talk with a field separator of /yes and then we tell it we want to print the number of fields minus 1 which is the second-to-last field that's where a number field number of fields is the last field so that will give us all of those object hashes so that's what this is doing so we take those two commands and these are wrapped in curly braces here right yep and those curly braces mean take these two separate commands and combine their output so we can put them into something else yes and we put them into a while read and that means take these kind of input put it into a variable called row and then we can use it somewhere else all the data collected from these to read I get we get all that in we exclude all the packing relation there we store all the information they've found and now we're ready to do something with it yeah this is just running a memory map yeah absolutely so what we do with all those object crashes is we pass them to get cut file so I'm so we use that earlier on looking at the Royal objects for the mix and things and that outputs all of the tax that's in that object so with a bit of luck we can make this file executable with chmod of change modes change the farm mode yeah to ads and execute there for this th ton and if we run it we should get an output that is all of the files all of the objects that has ever been in this repository and then we can grab right so before I grab for all of the functions yet we've got a little bit of an issue here where we've got this line binary file standard input matches yeah and what this means is grep thinks that we've given it a binary file which we totally craft because it was probably an image in there awesome thing none like non ASCII bytes like some no bikes probably it decided oh this is a binary file I'm gonna stop putting stuff I'm just gonna stop up and do it so you'd break yeah absolutely and I might be missing some information because of this right but grep - a stops that from happening so now where's my scene we had more yeah absolutely so if we look at the man page for graph and find the - a option we can see this process a binary file as if it were taxed this is equivalent to - just binary - files the sixth option or we can use - - text without the hey if I use WC or the word count function I can see again 162 results yeah but then with the - because it's stopped with the - a I go oh for sure so like if I was gripping for like AWS keys yeah I would have maybe missed or if there was important URLs so one of my favorite patterns I have the GF is the URLs which just finds every URLs and there and then I can go and grab for whatever my target is there's gonna be a lot in here right wait I don't know what we don't care about like the Facebook and upfront and stuff yep and there may be like a little bit funny like you've got semicolons on the end there's a first pass I'm I grab for say took line just that you be you are yeah quite a few in here a lot of them are going to be boring there's going to be duplicates and that kind of thing so that's 5 min to bend so we can play with so the food L win first thing we're going to do is sort them yep by unique we've got 700 of them and there's gonna be duplicate parts between these things so I'm gonna use another tool called unfurl that's another one of mine no more things unfurl can do is say give me the unique parts for example yeah so these are all of the unique paths that we show these are unique so we're not getting our duplicate and that's what we use dead what's a great workman yeah pretty much pretty much and this is a wordless bow like cats yeah but if we go back to where we were we might want to have a look for query string parameters right now first of all start to so I think it's called piece that's you keys so these are all the query string parameters the miner extension for example this would be a great source of of IDs for that and if you're just looking for URLs generally things with wayback URLs right yeah which is another tool yeah it goes to the archive or back machine yeah pulls out all the URLs that it knows about we'll just let's just pick on like example.com to make that a little bit faster yeah so that's gonna give us a whole bunch of URL back and we can pipe them through and use all of the different facilities so you can give us all the keys which is great for building web works for a miner yeah the values as well which are great if you're putting things into like intruder yep and building word lists for that so you want to try a whole different so if you undo your fussing yeah good way to start because you are also now creating constants that are think that's the big difference between this way of working and using just like the stuff that's inside class yeah like I use tech missed all the time it's great I love it but it could take a lot of time to run through your mega raft yeah absolutely I don't want to run like a hundred thousand paths when I could do that's my general approach for like approach doing reconnaissance on a target or any of this kind of work really where I have to go and fetch some data and then like do something with that thank you very much Tom that that was Wow if anyone of you out there when a get in contact or or get some more information on Tom where did they find you I'm on Twitter I'm on github Tom nomnom on my mum pretty much everywhere yeah no long calm cool epic VBAC

Channel: STÖK

Views: 184,247

Keywords: bugbounty, xxe, burp, bash, stok, stök, stokfredrik, ethical hacker, bug bounty, bug bounty for beginners, pentesting website, pentesting websites, pentesting, burp suite, recon, tomnomnom, tom hudson, vim, vim tutorial, bash tutorial, linux terminal, how to use vim, how to quit vim, linux command line, how to create a wordlist, how to do recon, hackerone, how to code, vim editor tutorial, penetration testing, hacking, bug hunting tutorial, web pentesting tutorial

Id: l8iXMgk2nnY

Length: 36min 16sec (2176 seconds)

Published: Sat Jun 22 2019