Mastering the Nmap Scripting Engine - Fyodor & David Fifield - Defcon 18

the nmap scripting engine is one of n Maps most powerful and flexible features it allows users to write and share simple scripts that they can use to automate networking tasks from enhanced network discovery to vulnerability detection or even exploitation and the good news is you don't have to write all these scripts yourself and map ships with a library of 131 NSE scripts which already can handle a wide variety of tasks I fill that NSE really completes and Maps scanning mission we already had host discovery where a map scans the networks and tries to find the machines that are up and available we had OS detection where n map would figure out what operating system those machines are running we had port scanning followed by version detection in order to find all the services there what application and version numbers they're running and then NSC is basically the glue that binds all of this knowledge together about the network you can use it to interrogate the host in pretty much any way you desire we're going to look at an example on this slide I don't know if you can read it in the back but we're going to go over what it says anyhow basically here we're doing a scan of a machine scan me on mapped org that's when I run for the purpose of people testing their nmap make sure it's working well and basically you'll find out in this talk that I'm not really shy about scanning other people's machines and so I feel it's only fair to provide one of my own that people can use to scan me back we're also giving the capital a option which tells M map to use its advanced features such as OS detection version detection and the map scripting engine and then we get these results now most of it is going to be familiar to typical M map users regular M map users but I want to highlight the M app scripting engine results from this first we have the under SSH the SSH host key script which just grabs the cryptographic key hash that the server uses to identify itself and while that might be useful for say a script that you would use to scan the network and find bogus keys that could be indicative of an attack or Trojan SSH what I like to use it for is machines as form of unique identification that is independent of IP address so say I'm scanning a machine I find one I'm really interested in to it maybe I manage to brute-force some passwords the next day I try to get into it further but it's gone maybe it uses a DHCP client it could have changed address to anything else well with SSA toast key what I do is scan the network again and try and find that key similarly if I'm doing a big scan and I see a bunch of machines that seem to be configured similarly are those really different machines or is it just one machine with a bunch of IP aliases well with host key script that's a good way to figure it out you'll also see some scripts below the HTTP port we've got HD HTML title which is a super simple script which just grabs the route web page from a web server and tells you the title very easy to do but useful information in the context of a scan we also have HTTP methods which basically says what methods does this server support get post whatever in this case it says the trace method is potentially a little bit risky here's a URL you can use to find out more if it had like the delete method or the put method then it would be sounding even bigger alarm bells so the nice thing is that with NSE it puts the results down here by the port or host that they related to so you can see them all in one place now I mentioned that we already have a library of scripts here is this zoomed enough that you guys can read it in the back yes good deal so this script we have 131 of them and I'm certainly not going to go through them all right now but I want to mention on the left side these categories that we have so you can get an idea of the breadth of asks that NSE can be useful for the off category is for authorization related scripts like we see up here the AFP brute force script which does brute force authentication cracking for the Apple filing protocol we have default scripts which is a hand curated set we select to find the ones that are most useful to people and least likely to cause network disruption or to be perceived as attack or otherwise undesirable we have discovery tests I mean that's kind of an maps bread and butter so if you click on a category you can see all of the scripts that are in that category and so for discovery we have a bunch of information if a db2 database is found tell us what's up there grab information from DHCP DNS HTTP and it goes on and on we have the das category for denial of service attacks there's only one of those I think we only have one in the exploits category for exploits now external that's kind of a policy related category fuzzers we have intrusive and safe categories are two important ones to know basically the safe ones we fill those are the ones you can run with little risk they're not usually going to use a lot of network resources they're not likely to crash servers they're not likely to be perceived as really hostile by a network administrator fully why is it intrusive and does that matter to me before you decide on your scan we have a malware category for example nmap was one of the first scanners to remotely detect the Conficker worm last year and the malware category is where that would belong version detection and maps version detection engine now has more than 6000 version detection signatures in it so we can detect a huge number of applications but there are some protocols that vary basically designed to hide from people trying to detect them on the network Skype is an example and for those where our version detection is in long enough well we found methods with NSE with its ability to interrogate multiple probes and analyze the responses more carefully is able to do those and then we have vulnerability detection related scripts so this NSE doc page which you can visit at any time a 10 Mac org dot NSE doc is really an easy way to find the scripts that you think are going to be useful and what you find when you find the script you want you can actually take it a little bit further we're going to look at a script real quickly called NFS - LS that basically says if nmap detects an NFS server on the network you can use NFS LS to say what shares are available what list the file names in those shares as well as the permission information and yes this is stuff you could already get from running other tools separately but the nice thing is to have it all in your scan results all run automatically together and so you find that through NSE doc and then you can actually click on it to get information about that script itself so the very first link is to get the source code of the script if you really want to figure out what it's doing that's where you're going to go the next is the description of the script basically tells you what the script does you get all the arguments for the script we have the wrong script here obviously this is the show mount script instead of the LS script this one obvious a bigger summary because it's more more useful of a script you can see the arguments it takes like maximum number of files you want it to show whether you want the modification time or access time or what you get example usage so you know how to use it we see some of the arguments are from a helper library we have for our PC and NFS so you can see these arguments aren't specific to this script but they're specific to a library that the script uses and so they might be useful you get an output and as you can see the author of this tried hard to make it look basically like a directory listing you'd get from LS and then you get things like what categories it's in in this case for listing NFS directories it's in discovery because you're figuring out information about the network and it's in safe because you're not really doing anything intrusive you're basically using this service as it was intended so we have a lot of scripts now 131 I mentioned and it's always growing here's an example of the data behind that growth we started out with about 20 scripts it took us some time to get our review process in place to get all the infrastructure there to easily write and process scripts but then lately it's just been growing like we can't hardly believe I mean we've been working for years on this system and had it in nmap for three and a half but more than half of our scripts have been written in the last year and I won't be surprised if when Def Con rolls around next year it's doubled again so now we've basically shown you the raw basics of NSE basically what it is what it can do now let's actually show you an example of what it can do and to do this I want to introduce you to a set of scripts written by a fellow named Ron Bo's who spent months researching the SMB and MS RPC protocols which is not something I would really wish on anyone but I'm glad he did it and these 13 scripts are a great gift to the nmap community basically there's informational scripts to just query the SMB server and say give us the detailed OS information including service pack and such the server stats the system info the security mode we have enumeration scripts to say contact the server and give us a list of users or domains or groups or processes sessions or shares and then we have three more intrusive scripts we have SMB brute which does brute force authentication cracking against the server to try and figure out passwords and it can work in conjunction with in conjunction with the SMB inu musers so that you already have a user list we have SMB check vulns which checks for a number of remotely exploitable ns our PC bugs and then we have s Mbps exact which allows you to execute arbitrary commands on the server with some canned ones available for you for things like dumping hash word passes dumping password hashes you can look at the different you might want to start a remote display server that sort of thing so Ron sent me all these scripts and I'm like these things are awesome but I don't run very many Windows machines on my home network and so I was thinking whoo how am I going to test these what sort of company might have a lot of these to run against so I thought you know it would be pretty interesting since its own their own darn protocol you would think if anyone can secure it Microsoft would be able to and I thought it might be interesting to see what sort of policies they take do they just ban it at the perimeter do they lock the servers down so that they don't allow the guests access and the IPC share and that sort of thing what is it that they really do so I designed a scan step one was finding the target IP addresses allocated to Microsoft and I use typical things you know look up the air and database to find allocations and eventually I found more than a million of them and decided to scan them all I started with a broad version detection scan so basically this scan I didn't want it to be super super intrusive and detailed for the first scan I kind of wanted to have an idea of what was there and then I could scan further on once I've seen the initial results that gives me faster results and makes it less likely to raise a lot of eyebrows so we say nmap to aggressive timing scan what we've empirically found to be the 50 most likely ports to be open do version detection OS detection do a minimum host group size of 128 which makes it faster because it scans more in parallel set a host time out of ten minutes so that we don't waste too much time on any single host send the output to files and I give it the list of IP addresses and this sort of scan not many years ago could have taken a month to complete and then if you wanted it any faster you would have to use a bunch of really detailed performance options to tune it so that it goes faster but not too fast that you get inaccurate results fortunately n Maps gotten a lot smarter in the last few years developed a lot of more clever algorithms and in this case we were able to scan the million IP addresses in about one day 26 hours and we have 74 thousand two hundred ninety three hosts up so let's take a quick look at the results here and I'll zoom this without all on the screen good and so basically this file is a big one and that's actually the problem with it I mean more than a third of a million lines this file is intimidating on its face and for us to go through all these it would basically be next DEFCON by the time we got it all done so I want to show you a little trick I like to use in those sorts of cases and so here's a scan or a simple UNIX command that I'm going to run which basically says take all of the open ports found during the scan and look at the different versions running and give me a reverse sorted list of how often every sort of service is running so we take a look at the results we see the highest is I is six and by the way I did give these results I did these scans last year and gave these results to nsrc so I hope hopefully they fixed it by now if not it's going to be some long nights in Redmond coming up we have AI is six so at least we can give them eating their own dog food here running some of their own stuff but rather than look at the top of the list what I like to do is go down near to the bottom and look at the more obscure services that they're running because those are the ones you're more likely to be able to find a little bug in I found little printers that I could get into the administration port on check their toner levels make sure everything's okay there various teleconferencing systems there's just all sorts of bizarre stuff on here but we're kind of getting distracted as much fun as it would be to take a voyeuristic view into Microsoft's Network right now it's really NSE and SMB that we're most interested in so let's take a look at those results so the vast majority of Microsoft's Network blocks I'm happy to say they basically filtered all of the MS RPC and SMB ports at their gateways and so that's something that enterprises and businesses should really take in mind although you should have already I mean if Microsoft feels that it should be blocked at the network and can't be really secured it's probably a good idea for other businesses to do the same but note that I said the vast majority of Microsoft's networks blocked these ports there were actually some that didn't and I found dozens of hosts that had port 445 the more modern of the ports available so from that I designed a new scan and basically I said nmap and the key new option we're going to look at is - - script equals to say run and map scripting engine and instead of doing the default scripts I showed you we're going to do SMB and Noom domains and Neumann processes basically all of the scripts that I showed you before that are in the less intrusive categories like the enumeration scripts the information-gathering scripts even I'm not crazy enough to do brute force authentication cracking and crack their passwords and then presented at Def Con but even from there we ran the scan and got the results and I would like to get some commitment from the audience here before we look at the results who here thinks Microsoft was totally secure huh we've got one person but I think there's probably several Microsoft people in the room who still aren't raising their hands but let's let's take a look maybe Microsoft will surprise us here so we're going to look at the results and we don't have time to go through all that many of them so I'm going to go through one of my favorite machines can you guys can you read this okay so we have 976 closed ports so that means they've defaulted the port's to be accessible so you can tell if they're open or closed rather than filtered and so instead of default deny and allow what you want they've default allow and they filtered a few ports like telnet and off that they specifically wanted to prohibit so that's the less secure way to do it so we'll deduct two points for that next we look at a Maps open port list and as you can see there are quite a few of them more than is really easy to secure and so for having that many ports open I think another two or three point deduction is warranted but the part we're really interested in is the host script results this is where our NSE action comes in first we have their shares to enumerate they have some sort of admin share a/c drive a D Drive now these ones are restricted shares so you need a password which you could get thanks to maybe our SMB brute script but one can argue that it's not best practice to share your hard drive over the Internet even with a password and so we're going to give them give them some points off for that maybe four or five now we're going to continue on and we get to the SMB II new Muser script which is when I talked about and here we get the list of user names on their domain controller and we have some of them are actually pretty interesting here's the administrator user this guy I like I took off some of the last names to give him a modicum of privacy but Richard his user name is the boss man which I thought was pretty cool let's see what else there are a few other ones that are worth pointing out here we've got this msb 50 net that's the building 50 networking team a nice thing about this script is that in the verbose mode - V it shows you all this extra information rather than just the list of user names there's a user not a guest exclamation point and then yeah there's the support user Microsoft Corporation redwoodm Washington but my favorite is the t-shirt ho so that's the sort of information that you can get by doing large-scale scans with this these SMB scripts and with NSC in general so as you can see there are a lot of fun I'd like to mention again that these scripts were actually written by a fellow named Ron Bo's and all I really did was take them and point them at Microsoft and pull the trigger so Ron really deserves most of the credit and I think if he was able to get in the room yeah there is yes thank you Ron these scripts are awesome we also have a fellow named Drazen Popovich who's been working on us this summer to improve our SMB stuff even further so now we've gone on how to use SMB scripts basically what they do but the next question is what about writing NSE scripts if you think about it using them is probably as much as your average casual end map user is likely to do but really any script Kitty could do that what do you do if you want a networking task which we don't already have a can script for what if we have a script but it doesn't behave in exactly the way that you believe it should this next section is going to show you how what we provide to let you go to the next step and even if you don't consider yourself a great programmer how you can build these scripts to run your own tasks and we're going to start basically with these scripts the language that they use is called Lua it's a great little language it's easy to learn if you know other languages like Python or Perl or compiled languages like C you can probably figure it out it's tiny to embed because we didn't want to bloat your end map the Lua book says the complete distribution source code manual plus binaries for some platforms fits comfortably on a floppy disk so for those of you young people in the audience a floppy disk it's a small storage technology now Lua is also widely used known in debugged so we weren't taking something brand-new that they might stop developing the next day it's been in use since 1993 it's best used known uses in the game industry with games like World of Warcraft or crisis but what we've been seeing more and more is that security tools have been picking it up we're talking in this talk about how nmap uses it for NSE Wireshark can use it now for protocol dissectors and the snort 3.0 beta has its own Lua interpreter to extend snort as well basically the only holdout now on the Lua bandwagon is probably Metasploit which is in Ruby but whenever I see HD more at these conferences I talk his ear off about why he should rewrite it yet again in Lua this time I think that's why he's been avoiding me the last couple of days it's also extensible we can hook it up to n maps fast parallel scripting engine it's safe and secure no buffer overflows format strings I mean Wireshark showed us the problem with doing their dissectors in C they got a lot of contributions but not all of them were secure and it seemed like every week you got a new vulnerability in a dissector that someone contributed its portable works on all the systems nmap does and it's interpreted so you don't have to keep recompiling nmap adds a few capabilities to that we have protocol and helper libraries they help you make say an SMB query or a HTTP request without getting bogged down in the details of the protocol for making those requests we have protocol group for sirs like we looked at SMB brute so that it can find the passwords for you in those times when you really need them we have SSL and we have the dependency system we talked about how SMB a new muse errs can then take that user name list an SMB brute knows to have users run fast take those results and crack them so that's a really quick overview of what we've added let's look at an actual example script so I think the script that I'm going to show you this time is called our PC info NSC and it basically does exactly what you would expect it nmap users and pen testers in general who've been around a long time you kind of get used to seeing port 111 RPC bind open so you just run our PC info - e hostname to see what's there and what RPC services are listening in what ports well it would be nice if nmap could do that for you whenever it detects the port open and it could put the results right there in your end map results so you don't have to keep it all separate and the nice thing is thanks to these protocol helper libraries and the like writing such a script took us it is 46 lines long so let's take a quick look at what it does we have a description we have a sample output these are for the NSA doc page I showed you we have a port rule tells you when it should run because it would be terribly inefficient if our PC info dot NSC was running against every port this says only run if it's port 111 or a version detection detects it as our PC bind and for either TCP or UDP protocols if that matches it runs this action the key line in this whole script is this one that says RPC helper RPC info it grabs that information if it failed it gives you an error message it sorts it into a pretty table sorts the results and prints it out to end Maps output and I'm sorry that was a real quick overview but as you can see you can do a script that actually does really useful things half of it is just documentation and the other half is pretty simple so I could go on with 131 of these scripts and show you them one by one but like I said before it seemed like it would be much fun more fun to have a live demonstration to show just how easy it is and so david is going to write one right in front of us that actually does useful things and execute it and hope that it works thank you David alright so to demonstrate how easy it is to write NSE scripts I set up a little example here and we're going to work our way through it anyone in this audience a programmer if you're if you're a programmer you can write NSE scripts so before I left home I set up a web cam pointing out my window at downtown Denver and I want to find it and see what's on the web cam picture the problem is I'm on a dynamic IP address so I don't know exactly where it is but we're going to write a script to find it before I left I found out some attributes of this web cam that can help us fingerprint it it uses TSU GBE to serve and it serves up a file called cam jpg so we're going to write a script that looks for those two things and finds my webcam so we're going to start let's call this HTTP webcam NSC sorry oh let's like redo this all right so the customer extension is NSC every script needs a description syntax highlighting sexy so this script is going to be in the safe category because we're not doing anything weird also I would put this in Discovery every script needs a rule in this case we're going to use something called a port rule which decides when the script is going to run and this in this case we just want it to run when the port number is 80 so this is real simple return port number equals equals 80 and then finally the action is where the script really does its work again it takes two parameters the host and the port now we're going to be using HTTP functions so we need to require that library this is just like an import in any other language let's define a local variable here for our response and we want to retrieve kam JPEG now let's just test what we got from the response here if response dot status we're just going to make sure that it worked and response status is not 404 and remember what else we're looking for response dot header we want to check that the server header exists just a minute and string dot match is a standard Lua function and we're going to match it against the pattern tht PD then it's easy to return results from an NSE script you just return a string in case we don't have any results we just return nothing and then the script isn't going to report anything so let's run it I'm going to add a couple of flags on here to make this a little faster so pay attention if you have trouble with the speedy or nmap scans turn off reverse name resolution turn off ping scanning we only want 480 only tell me about hosts with an open port we want to run HTTP web cam the address range I'm going to skin so the address range I'm going to scan is this one oh I gotcha okay thanks and we're going to scan these addresses let's write the results to a file in case that Scrolls off the screen and when you're testing your scripts for the first time you always want to add the debug flag in case your programmer who makes mistakes it will tell you where it'll give you a back trace to your syntax errors yeah right Oh No so what is that one of the virtues of a programmer is hubris all right there we go ah that's the problem we need to update the voice recognition okay so an map is fast let's look at the log there we go this is the address of the webcam 66.7 on one 71.5 and that's how easy this write an NSC script let's take a look at it huh okay so let's not let this scare us this is just an opportunity for us nmap developers to do what we do best which is scan I want to show you a script that I've been working on all right you follow me all right so let's go through this one real quick you get the structure is going to be familiar to you description guesses HTTP passwords categories I can't really with a good conscience put this in safe anymore put it in auth category we need the HTTP library again I this is another password this is our username password database library which all our brute scripts use and I'll talk to you a little bit more about that in a second port rule is exactly the same and the action is pretty simple just a few technical details here we have a user named iterator a password iterator we have a nested loop here we're getting kam dot jpg the only difference here is we're passing it a username and password and if we get a result that is not for o1 authentication required we're going to return username : password so let's run that scan it's going to be very similar just take off a few of this stuff some of this stuff we don't need and OH change the name of the script thank you and let er rip so the thing about password cracking is it tends to take a long time let me tell you a little bit about our username password database library whenever we're shipping databases with nmap we like to base them on real results you know it's easy to go download somebody's random password database but you don't know how good it is you don't really know where it came from you don't know if it's going to help you in your situation so we try to base this on hey that was too fast I'm not done talking about the password database I want to say that our password database is based on some real measured results from some like public information some data breaches password disclosures and things like that we've curated them and built them into a pretty decent list that you'll get if you download nmap so that's what we've used here so now in twenty five point nine zero seconds it has cracked the password username web password monkey that's an honor of the goon who introduced us let's give it a try and there we go all right so I had to this was a I made the script very small so it could be understood in a short time before I add this script to nmap there are a few changes I'm going to make the port rule has to be more generic you know it's lame to just match port 80 I want to match all the common HTTP ports and also any ports that version detection is found to be HTTP we have a library called short port that makes that very easy I would add script arguments so we don't have to hard code kam jpg you could just specify that on the command line add documentation for usage and output so it can go on our online documentation portal and finally this is very important I want it to be able to cache credentials in our registry so that other scripts that run later through our dependency system have access to these now let me turn it back over to Fyodor with some final notes all right we're going to just run through these last slides really quickly first of all we have a lot of stuff coming in NSE we have scripts in the queue we have an idea for scripts that can run before and actually accumulate targets so you could do like a zone transfer or a broadcast peeing or the like and a hand those targets straight to nmap we have Zen map NSE in a groove integration that was going to be on the coming soon slide but instead it's here because we actually put it in the repository just about three days ago David was working with a fellow named Kubek on some path to create this extra panel in Zen map which shows you all the scripts we have available what categories they're in the descriptions what arguments they can take and let's you set those so it's a nice feature if you don't have all the scripts memorized I wanted to mention credit where credit is due I wrote an map thirteen years ago and sometimes get way more of the credit than I deserve it's really an open source project in the true sense with kinds of contributors all over the world and this is just a list of the people who've written scripts for NSC which is just a subset of the contributor set as a whole final notes the slides are up now at insecure org slash presentations you've got to scroll down a little bit to get to the 2010 part here's the URLs for downloading end map the NSC doc court'll the system Docs and for QA we're going to take that right across the whole in 118 thank you very much

Channel: nmapvideos

Views: 28,438

Rating: 4.9866219 out of 5

Keywords: Nmap, Nmap Security Scanner, Fyodor, Gordon Lyon, David Fifield, security, hack, hacking, hackers, scan, scanning, networking, insecure, Nmap Scripting Engine, NSE, Defcon, tutorial

Id: M-Uq7YSfZ4I

Length: 38min 25sec (2305 seconds)

Published: Fri Jul 20 2012