How to Use Python to Spy on Your Friends: Web APIs, Recon ng, & OSINT

you all right everybody so let's go ahead and let's get this started I'll kick it off so BB King is going to be giving this presentation kind of a continuation of a theme the last webcast we had was job talking about Python which was I think horrifying for a lot of people whenever he's doing regular expressions on the fly and people like what could I have a kitten is like a hose you going to do is change this in the reddit and now we're going to continue that name and that creepy vibe and VB is going to be giving a presentation on how you can stop your friends and family using Python because that's that's what we do here we bring families together whether or not certain members know it or not and also we have a really really cool conference coming up Deadwood South Dakota we're doing wild west hack and fest the training will be the 25th to the 26 and the actual con is the 27th through the 28th it is in Deadwood it is over right about the same time as dead weird which is an awesome super cool Halloween party you guys should please check it out tickets are still available though we're selling them quickly we do expect that we're going to sell out as always and we have some amazing people we I got Dave Kennedy's coming we've got it's kotas we've got Mike core we've got a whole live on people we've got all Egypt music yeah Egypt music's secure ideas we have heaven Johnson coming Larry coming your Hall is coming ok I can't even I can't even name them anywhere there's so many so but they're all on the website they are all on our website so we're pretty excited is yes so please please please check it out yes that is John said on the cavalettis never notice today he fainted I got fired ha ha all right all right and I'll hand it oh we were going to talk a little bit about the Python class for sands as well do we have that on another slide baby there we go so sands does a Python class if you guys want to learn more about Python check out 573 and with that let's kick it over to BB BB please just take it away all right so yeah like John said we had a great talk about Python last week about log analysis and more kind of sensitive interest rate of types from jobs who teaches that class and that's a was it that was great this is another use of Python and open source intelligence so my goal here is to kind of get you comfortable with Python if you're not into it already but you know enough to get around I'm going to try to help you get to the next step where you can start playing with it and find your own projects have some fun with it I find if I think most people are like this if you get one of those gigantic books like have you seen a learning Python book that think O'Reilly book is like it's like three inches thick I would get to page sixty and lose interest so I like that picture that's on Twitter it's like most coding books like it's the equivalent of how to draw an owl and it's like draw two ovals and you draw the two ovals and then it says color the fill in the rest of yeah you're good there you go that's it that's coding right look at the story I heard about the big Python intro book is you know it's mostly white space okay get it Python joke all right so I'm gonna help you get your Python going if you know a little bit this will help you find something to work on to keep going we're going to talk about what OSINT is and we'll walk through an example I'll show you some recon and G stuff and we'll get moving so that you can leave this with something that you can take away with you so when I first started I did this talk similar to this one a year ago at a Python conference locally and I want to put it in people's head looks what suppose indicted don't know what those ins is so I asked the developers that I know on Twitter and they said they didn't know what I said what and I realized that this was a case of sometimes we talk bad and we don't realize that we're talking bad we use words that don't mean things to the people that were talking to so Python house 1 and this was the first thing that kind of got me a little I don't know built Python was kind of strange when I came across this concept of list comprehension because it's it's something to use all the time in Python but the meaning of this comprehension has nothing at all to do with either of the words that's in it well maybe lists but but comprehension is you have to dig for the dictionary to find an instance of the definition that corresponds to a Python actually does for it so you you're not familiar with Python and someone says how you should use this comprehension for that they're not going to follow you so most n't is kind of the same way it's it's jargon I guess I didn't realize that it was but it is so very quickly I think most of the folks here know what Posehn is it's open source intelligence intelligence is just information that comes from different sources those sources can be open or closed so Busan is open source intelligence intelligence so just some examples of sources that are considered to be open and things that are not so if you don't need special access or permissions or credentials or position with an agency to get access to it then it's kind of open source initially years ago this meant like newspaper articles in foreign countries now it means anything you can find on the internet and a lot of things that you think might not be open-source turn out to be after there's a breach of some kind so this is what we're looking for we're looking for information that's just available out there if you know where to look and what to look for so here's one example now I did this just the other day I wonder if there was anybody at SpaceX named Brian who had a site on on LinkedIn so so I searched for that with Bing Bing does the same search operators that Google does and this is the guy I found and they pixeled out the stuff just I mean you can find in the same way but I don't want to call attention to this guy if I don't need to so open source intelligence what might we do with this on just this one page you know LinkedIn gives you people also viewed over here and some folks make their the profiles publicly put lots of detail out there about what they're doing which is awesome for pen testers that are looking for information how we get more information from people so this guy was the manager of vehicle assembly and over here we have the VP vehicle engineering so maybe that's his boss we have other people in the guy named shaundra's of manufacturing Stephanie's a supplier performance specialist cesar down here and Kristin down all these people that he works with so maybe if you find his phone number or his email address and you want to try to extract some information about him this is some stuff that you can use to do that this is open source intelligence if you're interested in Python and open source intelligence where those two things get together you could do much worse than looking up just insights he's a he has a website called automating pollutants and he's very active in this he's he's he's he's all about teaching and he gives you really well documented code examples things that you can follow things you can build on this is just a recent thing from him about monitoring pastebin for terms that might be of interest to you so he walks you through setting up a script and a file of input terms to search for and you can have this thing run periodically on you know shared bps or you can learn locally and whenever those keywords show up in pastebin you'll know about it so you don't have to bother doing the work yourself once you set it up and what you're looking for this is when you make the computer for you and you just process the results another quick example before I get into the meat of this need a debit card on Twitter is kind of fun to follow this is again I pixelated these things out they were not like that when I found them people I don't know why but they like to post photographs of their credit cards or their drivers license or their first paycheck or their Passport crazy stuff that people just stick out there on Twitter maybe not realizing that it's public maybe not realizing that those numbers are meaningful to people and could be useful I kind of eat that but even there you know people play the game both sides so this one he's talking about the number who it's a lucky card because it's got 777 on both sides well it's the same number on both sides so of course it does and if you look at this one here see it's already cancelled and he's trolling you so you can't always trust the information you find but there's a lot of it out there to look at and it can be fun so so why would you use postings for your next Python project because it's kind of fun and it's really what makes it fun to me is that you get to see all this information that's already out there just kind of for the noticing it's a it's a whole different web out there once you start looking at it through programmatically through python or something else so here's one example that that we do network when we're trying to do some reconnaissance about a company will find individuals we look for email addresses we look for lists of people who are working at a place and try to find out what we can about them so that we can you know buddy-buddy up with them do some social engineering see if we can get some more information out of them so I started with them we're giving Python so I googled the phrase Python programming luminary and Steve Holden was like one of the top four hits and he's also got a website up here on O'Reilly and he's also publishers email address so I'm comfortable using his email address as an example because he stuck it out there for us so you start with the email address and you want to check to see if it's valid there's a service ethical male custard there's lots of ways to do this but there's one out there called melchester and you just give it an email address and click check address and it goes and uses the domain name it looks up the MX servers and then it queries those MX servers to find out if that's a valid account not just about email address like a regular expression like is the format correct but does this email address exist on this domain there's a couple ways you can do that this is all the background for you and it tells you whether it's valid or whether it's invalid so not Steve is invalid it doesn't exist on this server so you get one of two inches it's a good address or it's not address right that's great but there are some servers that don't give you that information so if you if you've ever interacted manually with a mail server which is which is fun you know there's like two or three ways you can do it you can type up there's a verify command that just says does this exist and if that's disabled it will tell you to just try to send an email and we'll see what happens and the other is you can do receipt to our cpt underscore two and then type in an address and it returned and it will sometimes tell you that that's not a valid email address if the verified one doesn't work and some of them just won't tell you some of them know what game you're playing and they'll say if you want to send an email you know send it on the bus again so Yahoo does that Gmail does that a bunch of servers are starting to now not tell you whether it's a valid email address so this is where you get into the ocean port and a learning part and the pay attention the advance paying attention I talked about earlier so imagine most of you know about f12 in your browser extended for years now so I think all the major browsers now do this on any page hit f12 when you open up the developer tools console and in was this program I think this is kind of what looks like these are all tabs different things you can look at and it will show you when you send the request it will show you what the request looks like so this request when I tried to validate Steve's email address it sent it to this URL it send it by a post not again and the response was 200 and these are the parameters that it sent it's a language which I didn't type in and an email which I did so the interesting thing here is that all I did was I said hey is Steve at home and web valid email address and my browser the web site added some information for me it said oh also we want this to be in English so this is a simple example of that but this this is how people get into trouble with with online services a lot of time is the the information that you intends to post is not the only information that gets posted here it's harmless with the the language but there are other cases where that that metadata that extra stuff can be super interesting there's a Justin Tyson thing on looking stalking people basically through Facebook and so this one for this one to work you have to be able to see someone's facebook timeline so maybe it's their public timeline maybe you're friends with them maybe you've tricked them into being your friend and you look at the timeline and buried in the HTML is not just the content that they're posting but also a timestamp of when everything is posted so what that person posts that post they're not typing in the time but the browser is adding that for them so this is again something you don't intentionally put but it's out there so so Justin takes that to to build like a histogram with when people are posting so you know when do they sleep when are they at home are they posting at work all day just kind of interesting things funny things but so how is that useful it can be usefully if you are on that you know if you're talking sometimes me of course but it can be useful if you're maybe a part of your job at work is to monitor your employees internet use and maybe they're not supposed to be posting on Facebook all day maybe you pay them to perform some tasks other than posting on Facebook and if there's an investigation you could do this you can check their Facebook Timeline and you can see when they're posting and you know if they were on the clock at that time well might have got some information more than just speculation about what they might have been doing when so this is what Milnes are doing if it's talking to the mail server you're not talking about server it is that's one of the assumptions in recon ng and a lot of open source intelligence as you're trying trying to make it so that the research can't get tied back to you so if you interact with a system that is controlled or available to the target then they can potentially notice that you're researching them that you're looking at their stuff and initially here you don't want to do that so mail tester is talking to the mail servers but you are not and I I suspect it would be difficult to go to mail tester and ask them hey who was it that looked up Steve at home web at you know 1:16 p.m. the other day I don't think they'd have that information so let's some look like so we figure out what is the URL we're getting to what's the method as opposed not yet and what are the parameters that go Python has awesome modules and this is what I think like the greatest value in Python is just a huge variety of really high quality modules that insulate you from lots of details and make things super simple to do so this is all it takes to send that exact same request in Python you the request module and you have to install first is not part of the default install but that's literally three words pip install requests in a second and there it is so this is the request you're sending we got this out of the browser and we're look at the response status code and it's a two hundred and but here's the response so so what is in that response it's the HTML that your browser this one surrender right it's not giving you just a quick answer whether you're good or not mail address is valid or not it's buried in there somewhere but it's not needed - fine - yep small the scroll bar is there's lots of stuff so how do you pull it out and this one this one kinda takes me back years and years ago before I use Python I use Perl and I still miss pearl but we I would try to parse documents for a lot of things at work and it was a lot of parsing HTML there was a Perl module called WWN eyes that did a lot of this stuff for you but it would choke on things that were not well formed and browsers like the fundamental purpose of a browser is to render poorly poking formed HTML so things that the browser can make sense of that it's difficult for modules to make sense of it's difficult for you to manually go in there and handle well there's no closing tag or everything's on one big long line or there's lots of white space between these things lots of low-level stuff that the request module can help you not worry about and other modules there's an XML parsing module I'll get to in a minute that helps you with that also so anyhow so regular expressions again right this is kind of a overkill for looking through the response and if you don't know more about regular expressions in point on jobs talk from the other day is fantastic for that I'll show you why this is a little bit dangerous just have this here and not a little thing in front anyhow this looks through the content for the phrase is space valid and here I'm printing out the zero for the first match that matched and found it so okay so great so it said it is balanced but maybe that string appears more than once in the response and if you look at response as rendered in the browser you probably not going to see it okay I think I showed you before is this page there's not much on the page but it's not uncommon at all for texts that has displayed to appear as in a comment somewhere in the HTML or as an attribute for an HTML tag with lots of places it could be yeah this may be this is part of the JavaScript that's totally unrelated to what you're doing so hey Brian you know just quick note it's kind of it kind of looks flaky on your screen with your share your screen but I think GoToWebinar is flaking out because your audio is going up and it's going down it's I think it's GoToWebinar but everybody that's on just so you guys know we will share the slides as well so even though it kind of looks durable you guys will get a copy of the slides to make it easier for everybody to see it after the fact all right I'm sorry about that I check my speed everything's good seems like it's good network wise so yes slides available so we could parse the document menu let's we just talked about go through and try to you know build the Dom tree and figured out yourself what if it's not well formed that can be typical if you're new to Python you might know about modules but there's lots of them so how do you know which one to start with and this can be a rabbit holes you're trying different modules and different philosophies you'll find with Python as with anything there are like religious zealots out there who will say you should never ever do it this way and people say you should only ever do it this way about the exact same thing so when you're getting started it's hard to know what to what to look with what to use what to pick up what's valuable and so the way I solved that problem for myself as I try to look at what other people have done before this is where recon and G comes in doing this looking at other people's code it can save you lots of time it can show you how to use features and techniques that you didn't know about maybe there's a tool that you use all the time but you didn't know what had this one cool feature here that's going to save you a lot of time that's that's how I learn now I don't look at books too much I try to look at actual living code to see how people actually do stuff so like I said recon ng is a tool that's been out there for quite some time now it's sponsored by the HS but Tim tones does all the working things it and it's out on its own excellent bitbucket not github but recon ng comm will send you there it's open source it's free there's there's not even like a paid version or paid modules it's all entirely free tim is really open to to new developments of additions and modifications he's got a development guide out there that I suggest you read before you send in a pull request because he's got some philosophy about how he maintains this if you think about open sourced up where you have contributors from all over the place the contributor is going to give their piece of code and then right away and the maintainer has to keep that going for however long so he's reasonably got some expectations and some requirements for how those things should be built so that he can maintain them over time the biggest thing there is he truly tries to minimize dependencies so there's a there's a parsing module in Python called beautifulsoup which i think is a play on tag souped like a things of digne sectional tanks tanks so beautiful super there's a lot of data that's have I talked about earlier where it's not a well-formed document maybe there's a missing a tag tag a beautiful suit can help with those things but you don't have that included and so if your pull request pulled in this whole other module he's going to say hey wait is look at maybe there's some other way you can do this at the dependency it's not part of the default install so anyway we the read that in the development guide before you do anything major on those things so in recon ng there's there's a bug that's just a database and then scripts that call third-party services like mail tester to populate that database these are the categories the tables that are in the database and these are some of the modules for how you can fill them here's the schema for how things are laid out the contact is a person a profile is like your LinkedIn or your github page repository is your github repository company is credentials this is from public data leaks and hosts which is hosts web servers later kind of servers after that we can find through there so getting to use recon ng at first I first started using it I had the hardest time making sense of what the syntax was for calling things and the the biggest the thing that helped me the most was to realize to see this this separation this is how that modules are published it's kind of like my display it's meant to be a little bit like Metasploit so there's a section and slash and then from - from what table - what table and then the module name so the big LinkedIn cache module is part of the Recon family it pulls from the company's table and it populates the contacts table so that makes sense right so in order to use that one you have to have a company and when it's done it will hopefully have populated the contacts table a little bit here's one that takes contact and updates contact so this milchester reach from your contacts table get some more information hopefully and then updates the contacts table with that and this this was the trick this was for me this was the thing that made recon ng easy to use going from I don't understand this to really easy to use you can search for anything that's part of that path so if you search for contact - this is going to find you any module that starts with the contacts table and does something so you could also search for - contacts and that would get you things that fill the contacts table so we use milk tester manually and there's a module from real testers so this is the great thing to start learning from because you know how it works on the web and you know how it works in you know a little Python thing there at least part of you not to make the request and get the response doing so we're going to search for mail and it turns out there's only one thing I had mailed in the name of it and show options or select Metasploit there's show info which gives you options plus extra stuff there are choices for the source all of these all the modules you can there's a default source usually it reads from one of the tables but you can feed it stream directly change the query that it runs you can control where it gets its data from when it runs the runs the module so here's how to manually do this guide that we would add to the context table and it prompts you for all these things so I filled in his name and his email address to do all optional nothing is required in the Recon entry database and this is what you end up with so looking at the options the source is misusing the default source and the default years term is I'm sorry not to remove invalid email addresses so if male tester comes back and says that's not a valid email address you have an option here to tell recon J to get out of the database because it's no good you run the module and it gives you email addresses valid so it did all that work for you it created a request but sent the request and read the response that parts of the response and it decided whether it was valid or not all for you here so that's great but what is it doing we're trying to learn how how Python might do this what's it doing there's a turn on debug output in recon energy which is called this not called debug it called verbosity there's a verbosity level of zero one or two I think two turns this on since you what the request looks like so you can see same thing sending test mail that PHP to Mel Fisher comm with Stephen Holden web and the languages English and that's the response we got you can also set up a local proxy you can tell recon energy to use a proxy and I always send everything through virtually just because even what I saw just there isn't enough for me I wonder exactly going on so you can clean everything through perks me turn off interception and just go back and look at it after it's done to see what it's been getting so to see how it's actually doing that you can look in the module itself and here we are this is the recon energy folder that gets created when you check it out and then there's the folder called modules and then recon and contacts contacts and then mill testers so this is exactly what you use to load the module it's the exact same thing so once you go what you you know where on your file system to find that module and this is the entire script the whole thing is 35 lines long and a bunch of it is essentially comments so this I think is a good example of Python coding especially if you're just learning how to do this to me if something is concise and readable that's that's a win that's that's what I want to use as my examples to learn from so let's go through this module real quickly and see what's in there it imports two modules one is part of the recon module and the other one is El XML dot HTML and it imports a method from there so what is that what is that XML about HTML so Google for it to find out this is the page where where it describes that I just Google for Python L XML and I found there's this from string method which is what we're embedded in here hey BB we just had a good question for Mike you said can you change the user agent yes you can't change the user agent the the default user agent identified as recon ng as recon ng but it's it's a setting and you can set it to be anything you want so if you would like to be a browser you can be a browser if you would like to use your own user agent you can do that too so yeah if you want to be extra stealthy so that's you know send something from firefox to burp speak and see what its user agent string is and then set except that recon energy to use that same thing good question so this chrome skewing method takes a string and it creates some sort of a data structure out of it and that's good enough for now we got me to spend a whole a time going into the mechanics of that because the modules to make it for us that's the whole point so looking through this line 29 this is so this is from string has created some some object and then I'm not sure what's doing here it's removing something maybe that's important maybe it's not but on 29 this looks like it's getting a message list and it's using XPath and maybe you don't know XPath but if you read it it's kind of intuitive kind of intuitive so there's and looking at the last thing there and there's a table role in the last thing there so it kind of makes sense if you just take the time to look at it and try to move without what they're telling you hey Brian yes someone wants to know are you using Python 2.7 or three this is to seven the whole recovery project is based on the to seven branch the Python 2 versus placed on three thing is I don't know it's almost a distraction that's one of those things that some people have really strong opinions about one or the other and there's a good argument to use Python 3 and that is it's newer and Python - I think it's end of life is scheduled for 2020 but they're they're really like almost different languages you can't always take a Python to script and run it in a Python 3 interpreter there were some like breaking changes that they made on purpose for good reasons that make it so that you kind of have to choose - 7 is kind of cool because it back ported a bunch of those changes so that you can you can write a script that will run in - 7 and is more likely to run in a 3 on the 3 branch by 3 kind ng is all in - 7 - 7 is what I have that works so we're going to see what this is doing we're going to save those HTML files because then we can read them locally not to interact with the server it takes a lot of variables out of the equation and we're going to look through it to see if we can find what it was finding and this is a good excuse this is really just an excuse so that to learn some more tools everybody knows crap but what does the graph look like what you usually do maybe this isn't what you usually do and here's some stuff that grep can do that maybe will be useful in the future you can have a print a line number you can have it print some context around the match and not just the match so we're using that here so we're going to look for the at the last instance of the table and it finds it down here on line 149 so this is the last table row and then we're looking for the last table data and that's obviously below this one that's down here on where is it this guy and look we're getting to make sense and this is the content of that role so this make sense this is the text the string that we saw in the UI when we first used the thing through the website now how do you convince yourself yes here all right I feel like I'm interrupting you but another question which got these online program converters they translate from one programming language to another I hang about you say Airy but that's that carry that sounds horrifying I mean like and actually but it actually sounds like it might be kind of fun to play with I don't know if I do anything with like production level code it sounds like a neat trick all right notice some of the things are kind of deterministic like one of the changes from Python 2 to 3 is that the print function became a function so in Python 2 you do print and then your string and then python 3 do prints and then your string in parenthesis so that's an easy thing you could programmatically do you could just enter closed parenthesis but there's other stuff like the map that it does has changed I think the division operator I don't know what the difference is but I know that it used to return something I think it returned the floor like the lowest the the decimal part of the response or the integer part of the response and now some of it not deterministic so we go to local files we've saved that special responsibility response and then we still see that not allowed response because I don't want to do with that that's something that's you know it's a third option here it's good or it's bad or what do you think the module does with that third option what would you do with that third option is it going to be meaningful so we've load from a file not from a string that we've gotten off the liar's so we've got to read the documents a little bit here from L XML and there's a partial function that takes a file so awesome we're going to use that and in the interactive Python interpreter we're going to play with these to see if we can get the same kind of result that the the the Recon ng module got so then this is all it takes to so you open up the Python IDE title title I tell me what you call this thing the repple that's what is repl which stands for something so you import the parse function you parse the document that you loaded this is just copied and pasted from the ricotta and G and then message us to print it out it gives us that string got it and then there's this other line in the module that joins on spaces everything in the message list so okay so I'm going to print that and I get the same thing so this was a an array of one item so it's the same here I had more than one item I would have more than this here I'm not sure how that would come into play how that would factor into this particular service but there's some logic going on here that seems to be extraneous and then we'd sing with that one it was not found and this gives us the same kind of response that we'd expected so making sense and in the module the not allowed condition where the mail server is not going to tell you if it's good or bad just kind of gets ignored it doesn't it does not exist it doesn't exist we delete it if it were set to delete the email but the Madeline condition there's no case to handle that in the in the Recon ng module so so why is that interesting that's interesting because the module is doing stuff for you it's taking all of that big HTML response and it's pulling out what's important and it's doing something with what's important but the person who wrote this module decided what was important and decided what to do with it so in this case it seems a reasonable things do if it can't verify whether it's good or not it seems reasonable to just move on to the next one but maybe for your use case there's a better answer maybe maybe the mail server this you're using should always return good or bad and if it returns I'm not telling you that indicates some problems somewhere else so so this is to me this is like the difference between just using a tool and knowing what it's doing and being able to adapt it to different circumstances understanding what it does and why it does those things and then choosing for yourself whether you agree that those are good things to do or not that gets you to the next step where you can make some of your own tools make your own decisions and make improvements to things so I covered this we talked before what's most the extra why is why is it joining those things together on on the space string maybe there's something else going on maybe there's a different kind of response that has what's going to give us more than one an array that's more than one helmet long I don't know they haven't found out in a million's for this one but if you're interested if you just can't let it go this might be something that would be worth looking for maybe maybe male tester allows you to submit to email addresses at once I don't know so we talked about how to contribute to recon ng it is actively maintained and updates are very much welcome so this is one way to use Python to pull information off the web and this is like the complicated way the kind of painful way where you're pulling it out of HTML out of a context that has a totally different purpose it's meant to be rendered by a browser not to be picked apart by by you and your scripts and so if you can find a pis that use the services you want this is so much easier they the API is they generally send in return JSON JavaScript object notation so it's it's meant to be computer readable you can actually take that response and turn it directly into an object in Python and then look through it so the response is going to contain an array of things or a dictionary of things and you can look things up by their positions in the array or by their name at the dictionary and it's so much easier it's so much more reliable so the next thing we can do now that we know his address is good we can find out where else this guy is and this is where the spying stuff comes in if if this is a good email address then maybe it's used for other places and lots of services will use your email address as your username and some of them even use that in the URL like Twitter Gus that your Twitter handle is new intercom slash your Twitter handle github is the same way lots of places use that as your identifier so if you know someone's identifier you can go across all of those services that you know about and see if that identifier exists on all those services so you can see where else this person might be active again it's not like totally reliable because anybody can register any name but it's a good first step and it's a good place to go to look and see you know maybe manually verify if this is the same person on this other service or not so full contact does this for you they have a huge list of services and if you give them a yuning they will go and look through all those other services to see if that username exists there as well the website does this all on once the response is its it fills up as it finds answers to things so there's a big long list of services looking for and they turn red or green as the response comes back in so they have an API so you need an API key to run their stuff and this is this was what I would do to figure out how to use that API and pull out what's interesting to me I would fling some requests manually and by manually it depends on how the service is set up some of them rest-based API is the thing you're looking for is encoded in the path so those you can often use just through your browser you can type out the URL like github on github comm slash whatever for that username and if you get a response that has content than that it can that unit exists there you can get different response than it doesn't and then they do the same thing but with Python to send some requests just like we did before use that request module send the request parse out the response see if you can figure out what's in there that's interesting to you and this is where burke comes in so handy because with the script that you write is only going to show you what you've told it to show you and until you know how its formatted in the response you actually want to ask for the API documents should tell you that and they do but sometimes it's easier to see it in context with a real response so if you send all this stuff through burp you can see the whole response looks like and you can more easily know what you want to look for in that response there's a for the stuff that uses JSON there's a burp app store plugin called JSON beautifier which takes JSON that may not be formatted in a way that's easy to read maybe trending online and it formats it indented as you would expect it to be so that you can read it work easily so is an awesome helper for this as well and then the other thing you can do is run the full contact module from recon energy and with verbosity set to two so you can see what it's sending and what it's getting back the you know every eight minutes that I've seen the this was a full contact one that github on the digital ocean one all the ones I've looked at they have examples for how to make these calls using curl and this will save you some coding time also the command lines kind of get really long sometimes but it takes away a lot of complexity it focuses directly on interacting with service and not so much with the Python or whatever programming language you're using or the environment you're running from or all of that other stuff if you can do it with curl then you can do it somewhere else as well so using that with mr. Holden we found all this information about him just from full contact so the full contact took us an email address that's all it took and it looked up his profile in where was this this is probably odd I'm holding web and then it looked found him on github on Twitter on Flickr all these different places this one's interesting cuz Google+ here this is not his username this is not his email address so it found something that maybe he wouldn't have been able to find it quite so easily and then down here don't overlook this the competence at the bottom is you know it's just a number eighty-seven isn't different from 82 or 91 it's it's just a reminder that that this guy might not be the same guy it's just the same name somewhere so take all this stuff as possibilities and then verify those things especially if you're going to mess around with them like if it's a friend of yours and you're going to try to troll them on your other service do something to make sure it's them first make sure that you're not calling random people that won't know who you are it's it's just a starting point so in recon oMG these are all these tables so so now we've got some profiles for mr. Holden he's going to be filled in here with the what service it was that you are already that all that stuff so you can use all that's out of every kind energy directly to find where he is it will give you a list of the URLs in the Colombo's URLs and go view those things so it's the value in the recon energy here is that it does the work for you and it collects all the results in a way that's easy for you to follow up on but you have to follow up on them just because it shows up in results doesn't mean it's reliable so that's kind of the overview and this is where you can have fun with some Python and to see how the web works when you're not using a browser just some ideas to do pick some friends people you know maybe people you follow on Twitter or somewhere look them up through the API is that that service offers and see what you can find about them the Twitter API you can pull down somebody's lists of tweets all the ones that are available there's a limit to how many come down to Kindle it is with that it's it's the easiest way to get everything somebody has said on Twitter because the web UI doesnt show you everything in order you know some replies don't show up the same way as individual tweets to and things get manipulated and algorithms into different orders and that kind of stuff so that's a great one to start with the Twitter API you do need a API key but it's free there's limits rate limits to how much you can send at a given time but they're totally reasonable for this kind of stuff some practical things to do look up your employer or someone you might want to work for and see what you can find out about them if you can if you can establish a common interest with somebody they light your employer I don't like where this is going this is a great way to find a job this is a great way to to find that maybe maybe people that already work in your company that could help you in some of your goals so finding common interests I think it can be a little creepy if you don't do it right but just to find out that oh this other guy who works in this at the department I'm interested in he coaches peewee soccer too and and I've done that that's something we have in common so maybe that's an icebreaker some way to start that conversation let's see and then don't don't forget to go in there and look at the Recon and resources is really very consistently clean and solid to see how that stuff works so that you can learn how to do some of these on your own in this different use case now like recon energy doesn't have a give me a list of some of these tweets function because that's not its focus but maybe you want that maybe there's something you could write based on what's in there learning some of the cleanest of the code and how those things work what modules work that's a great thing to do I think and then write your own version and compare it to how the Recon energy one works if you can write your own briefing first and then go back and compare it that's I think the best way to do it because then you solve the problems yourself and now you go back and you see how somebody else solved those same problems so if you both did it the same way that's that's great for you know in one way if you did it differently that's I think even better because now you're going to see another way of approaching the same problem if you find something to improve and sometimes these things are stupid simple things to improve my first contribution to recon energy was I fixed a typo there was some variable and that it was portal and it was failing in certain circumstances because in one place in the code that didn't always get to it was singular so literally my first four requests for every kind of G was had an S so nothing is too small nothing was too dumb if it actually makes an improvement and then I had some resources for you to follow up on after this there's Rico in G obviously if Justin sites that he's a automating Oh since he's got some clients he's got a lot of free material available he's really really very good very friendly very good at what he does very generous in what he shares he also has those the two python books are from him so if you're familiar with those books if you've learned from those than they already know his style Michael Hoffman is that ocean ninja he's also at web breacher calm he's a sans instructor I think he's actually working on a recent course that's all I know about that but he also he teaches the web that pentesting course also very friendly guy very willing to help if you have some questions or pliers take a look at his stuff maybe you can pick up some stuff from there and in that last one Intel techniques comm is starting to I will set cross a line they're starting to be a little bit different focus this one is more focused on investigations like law enforcement type stuff like we need to find where this guy is type things more focus on finding individuals and tracking an individual's activity online so not quite the same but maybe maybe useful and certainly lots of the techniques that are that are shown there are great there's there's a book he's got available there's a you've got a meta stork engine in there that you can search the k-member how many it is a dozen two dozen different different engines for the same strings all at once you put it in once on this side a put the button and again his site doesn't searching for you so you're you're insulating yourself a little bit from the targets of those things but just pick something something to play with and and make yourself do it in Python and see where it gets you I think it's I think it's a lot of fun there really is a lot of fun that's that's where I end I have some time for questions or what any questions so thank you that was a good tip on looking at perspective employers someone's looking for a job I'm check out the features but what about PG pts as this is unique intensity follow different aliases I think what we usually do correct me if I'm wrong baby but whenever you go to like Keys MIT net for PGP Keys you can have multiple email addresses associated with a single key and that can lead you into additional like possible email addresses and profiles for target yes that's true I believe that that's where the modules in recon to give it an email address it will look it up on that key servers at MIT and also in like who is records on DNS I'll kind of set that back yeah pre hiring a BH is I oh no I'm not really good people sometimes you know they fall to the trees and and we picked them up as quickly as possible he would recount I like the what was what was so the total password count I think we had seven I think have seven people actually out the past seven off the right shoulder of BB right here summer 2017 so CI we had a couple of people that were interested that we did get another good question what made you move to Pike pearl I'm not even going to touch maybe take that one away have fun it was the proliferation of tools there in Python I would very grudgingly move to Python I was one of those religious Perl people and as soon as I saw that white space was significant in Python I went that's not for me but I got over it because it's really really effective and there's so many tools and so many resources out there that make it easy to learn and make it so valuable you know how Pomerance does a great quote on and he said his biggest complaint the reason why he likes Perl is he can look at somebody's proko and very quickly make determinate whether or not it's crappy pearl or good pearl he said the problem with Python because of the whitespace way indentations are held as all the code looks good even though it might be really bad code it's formatted very very clean and he says it makes a lot harder to find out somebody's a bad programmer of with AIDS Python so that's an interesting perspective - wow that's the first time I've heard that Perl code is no no no Python is clear yeah yeah pool code is it just wanted more consistent regardless of how good the code is true cool so probably kind of the same thing Python did around the time I switched over they started the Perl 6 project and so paral 6 is a break with Perl 5 - at least the same degree that Python 3 is - Python 2 so at some point I was learning any language anyway so might as well use the one that's everybody else is using cool deal alright so we're going to wrap this up thank you very much for attending everybody and as always we will post the video on our blog and we will shoot that out here in a little while so thanks again and we'll see you guys on the next webcast take care thanks

Channel: Black Hills Information Security

Views: 17,766

Keywords: Python, OSINT, Web APIs, recon ng, BHIS Webcast

Id: BOjz7NfsLpA

Length: 52min 18sec (3138 seconds)

Published: Thu May 25 2017