Defcon 21 - How my Botnet Purchased Millions of Dollars in Cars and Defeated the Russian Hackers

thank you for coming to my talk it's always a treat to be able to do this I've had the opportunity to do a lot of really cool things in my career and with bots but the one thing that gave me more satisfaction than anything else I've ever done is the time I wrote a botnet that purchased millions of dollars worth of cars and defeated the Russian hackers so let's have some fun with this alright I'm going to tell you a story that involves hacking it involves cars I like cars it involves Russian hackers which is pretty cool and more than anything else that involves screwing with the system thank you thank you or as I like to tell my mother creating competitive advantages for clients that's important it's easier to get a loan that way too so I've been writing BOTS for since about 95 started out doing remote medicine BOTS if you can believe that I've been involved with privacy fraud detection private investigations have done work for foreign governments and I've got a fair amount of my business that is with automotive clients what makes me a little bit different than I mean a lot of people write BOTS what makes me a little different is I actually talk about it unfortunately the only projects I get to talk about are things that are in-house projects that I've been doing it's really rare that I get a chance to talk about a specific project that I've done for a client but I got permission to talk about this one and it came about largely because when my last book was done this one through no start depressed by the way they approached no starch a Linux magazine and they said you know can Mike read an article for us and I really didn't have anything ready to write for them so I approached this old client I said you know enough time has passed it's been like six years let me write about this for a change and they agreed to let me do this but that's really because when you've got a piece of technology that provides a competitive advantage or allows you to screw it the system strategically you don't want to tell people about it right because that's your it's a trade secret really so if you want to get a little bit different view of this project if you can pick up one of the old copies of Linux magazine I write about it in a little bit different way than the way I'm presenting it here tonight okay what are you going to learn you're going to learn what makes a good bot project I'm going to have to give you a little bit of insight in how retail automotive works in order for this whole thing to make sense you're going to get an awareness of commercial BOTS and botnets and they actually do exist and I'm also going to talk a little bit about if I were to do this again today how would I do this differently because keep in mind this happened like six seven years ago so what makes a good bot project the very first thing you need to know is that you cannot be afraid to do something different okay if your company has an internet strategy assuming it has an internet strategy that just involves browsers and things you can do with a browser you're really missing out because you got the whole big wide internet available to you and everybody uses the same tool the browser right to access it and if you expand your scope a little bit and do things outside of the way browsers work or do things outside of the way websites are presented to you you can create a lot of really cool things okay don't assume just just a reason here how many people here have written a screen scraper okay cool how many people have written a spider Wow cool cool well just if you've got a client make sure they realize that just because you know how to scrape screens if you write a spider it doesn't mean you can make a copy of the Internet okay and you'd be surprised I get people approaching me all the time with ideas for projects a lot of them basically want to create a copy of the Internet so if your project requires both batch processing and real-time results you've got a problem or if you've got a project that requires just ridiculous scaling you've got a problem because unless you've got one of these your project is going to fail you know you're not going to replicate Google unless you've got one of these and then I tell clients after I say you know you really can't do this it's like why not and I'll say well because Google spends about a million dollars a day on electricity that's why that's why your projects going to fail realize that you don't own our refer to targets as the the subject server don't assume that you own that server okay for example I had a potential client approached me a few years ago and he wanted to monitor prices on Amazon about a hundred thousand I for about a hundred thousand items I thought that sounds really like a useful thing to do this guy was a big-time Amazon seller until I found out that he wanted to do this every five seconds you know as that's not going to work it's not going to work for lots of reasons if you did something like this Amazon would actually have to build additional infrastructure to support your your project and you'd end up in court with what they call a trespass to chattels suit and you want to avoid that it's very illegal okay number four and this is maybe the most important thing you have to have a realistic profit model you notice I'm saying profit model and not business model why do I say that this is why okay and if I'm showing my age here a little bit you can look at these my space actually made the list twice I think that's pretty impressive that's that staying power so why is it important that you have a realistic profit model you know why is it that when people approach me and they want to do something that could just as easily be done on eBay for example it this is important because the developer has to get paid okay that's very important okay about automotive retailing just a little bit here without this the project doesn't make sense new car sales are not as profitable as people think they are even if you can buy and service with that because it's incredibly capital intensive and it's super super competitive but you need to have new car sales so you've got credibility if you want to sell used cars this is particularly true if you want to sell high-end used cars nobody wants to go to the corner you lot for that kind of stuff the thing that I learned and I didn't realize I just assumed that all the used cars on a car lot were all trade-ins well that's not the case and it can't be the case because you can't grow a business if you're going to do that right and its really limiting car dealerships spend tons of money acquiring good used cars to put on the car lot and it's kind of bizarre the way it works because you walk into a carla and you know what the price should be for a particular car because it's very well documented right you can go to Kelly Blue Book or any place so dealers don't have a lot of space to work on on the price the final retail price but down on the wholesale side that's where the profits and that's where the margins are made if you're good at buying things for a great price that's how you make money with used cars and that's what this project is about so a car dealer came to me he had this great opportunity fund this wonderful website it was part of the the national franchise they were getting in a used rental car two years old twelve to sixteen thousand miles perfect cars that you'd want to have on your lot okay well-maintained unfortunately there was a lot of competition for these cars because all the people in that dealership chain wanted the same cars and the website was horrible and made it almost impossible to buy the cars so there's a lot of frustration this is kind of the way it worked there would be maybe two to three hundred cars presented every day and the cars would have little display ads like this that give a little bit of a description and there was an inactive Buy Now button okay again at exactly sale time that button would appear okay but the problem with this was it wasn't using Ajax or anything you had to physically sit and refresh the browser constantly to get that button to appear well this led to another problem and that there was incredible server lag my client and I think he was probably a pretty typical of all of them in this business chain he would grab every person he could find people out of parts out of you know off the sales floor administrative assistants he'd sent them all in front of computers and each one of them was assigned maybe about six cars so they'd have six browser windows open and they're all sitting there frantically hitting the refresh button constantly so if you think about this okay so this would have been roughly the equivalent of 36 users for this one dealership I don't know maybe there were 750 dealers that were doing this so that was almost 30,000 simultaneous downloads that were happening at sale time and what made this worse I mean servers should be able to handle that right but I think there was some inefficiency with the database possibly some bad queries are being made and this caused a ridiculous peak in server lag time right at the point where you don't want to have it and it would take you know wouldn't be unusual for it to take fifteen or thirty seconds for the screen to refresh at sale time sometimes it would just timeout so this was a real problem the other problem is that out of these say 200 cars they're up for sale every day there were maybe five that every single dealership in the country wanted either because they were the right color probably because they were a really great price or for whatever reason I don't know but every dealership would want these five cars so he had a lot of competition for the same cars plus server lag bad web design had to involve a lot of people to do this so this particular client I had written a number of bots for him in the past and he gave me a call and said can he help me out Mike so let's take a look so the problems where as if the system was way too manual to begin with so the way this would work he would have to manually go and select the cars that he wanted to buy he'd have to distribute the VIN numbers to the various people he'd have to call people in off of their normal duties that they would be doing they'd be dedicating probably a good 15 to 20 minutes hitting the refresh button every day so that wasn't good plus the Buy button took way too long to appear because of the server like so we came up with we ended up with two solutions one of them because work the second one because we had competition so let's look at phase one first here and again this is not like classic bot design and keep in mind this was done like six years ago so I don't develop like this anymore okay so here's what I did I came up with a web interface for my client and if you look here this is basically just for HTML frames that were independent from each other and you know they could just go to a URL pull this up and by the way I say botnet but this was all done on computers that we controlled well not controlled we owned okay there's a difference right affect all of the bots that I write they're all commercial BOTS we own all the hardware okay I just want to let you guys know that so instead of hauling in all these people hit the refresh button constantly while they should be doing something else my client was able to pull up something like this and quite frequently he would have two or three computers set up with this in the browser and he would just select what cars he wanted the first step was to log on they had several accounts for this it was a closed sale basically and they had several accounts they could use so the first thing they would do is they would pick which account they wanted to use for this particular bot and the next step was you would pick the VIN number of the car you wanted and it would go ahead and it would validate that that was an actual car for sale that's important because any time you're writing a bot you don't want to do something that could not possible to be done by a human and if there's a car that says is not available for sale you don't want to try to buy that because some system admin somewhere is going to say how did they do that what was that IP address where they're generating a lot of traffic really good traffic so it's important to validate stuff like that so as soon as the VIN was validated a little start button would appear so instead of being you know right on time when the sale was you could do this hours in advance get the start button and then I would start to countdown now the way it would do this is it was basically synchronizing its clock with the server clock of the sale server and this was really simple stuff in the meta refresh and the HTML meta refresh it would just start refreshing every so often and it would get you know as the sale got closer and closer it would refresh more often until right at the end it was like great lockstep with the server clock and as soon as it timed out it would go ahead and it would attempt to purchase the car now this shows just one bot client and basically the bot clients acted as triggers for the server that actually made the purchase and there may have been sixteen to thirty of these BOTS running or triggering the server sometimes we'd miss one but more often the sale was successful and we would send an email confirmation to my client saying you bought this car and we would also arrange for financing for him and while we were at it we'd made sure that the car actually was shipped correctly back to his dealership so the black provided a lot of utility in that regard so how successful whirring well before he wasn't getting anything and this was really frustrating for him because these were cars he really wanted and he knew he could make a profit on them given the price service they were selling for after we were getting probably about 95 to 97 percent of the cars he was trying to buy so the difference was phenomenal there was so much fun because even after I was done developing this I would get a call every day for my client 15 to 20 minutes after the sale and he would say Mike we bought five out of six today we got seven out of seven we got nine out of twelve and like settled down don't don't get greedy here because you know don't kill the Golden Goose so why were we successful at this well the main problem with the old one is that people had to wait for that stupid refresh button or that that Buy It Now button to happen and there was so much problem so much server lag that that was the problem and usually the whoever got the buy button first was the person that bought the car so basically what we did is we got rid of the buy button we just got rid of it and we replaced it with a timer that was automated so he didn't need that person hitting refresh all the time and it would just know what time to buy the car and it would go ahead and buy it the this type of a a bot is typically called a sniper you've ever heard that term before and I remember back in the day when I was doing this we were testing and I was going to write him an email that said something to the effect of I've got six snipers waiting to hit cars at noon hopefully we'll make some hits today or we'll have some kills or something like that and I was just about ready to send that email and I started thinking about carnivore you know and some of the stuff that was happening back then and I thought now just give him a call today I would never send an email like that never I'm not even sure I'd make a phone call so yeah watch your language okay so everything worked great for about six months and all of a sudden things weren't as rosy anymore we started not you know micro-cut was scuse me uh the client would call and you would say you know we only got two out of seven today um something's wrong and he did some research and he discovered through his connections he's got lots of connections that there was a group of Russian hackers that were hired to write a competing bot and they were someplace out in New Jersey or the dealership was out in New Jersey or something excuse me who what Yanam Noga Pony mile I don't know no comprendo so competition is good right that leads to innovation that was kind of thinking yeah let's this is going to be fun now we've got an arms race going on here so here's part two of the solution what I did differently is while I was synchronizing clocks with the sale server I started looking at lag time and I got to the point where I got really good at estimating lag time there would be at the sale time so in other words what I was essentially doing is I was estimating how many users around the system and with that information I would not set one attempt to buy the car but for each bot I would launch maybe between I forget what the real number was was I haven't looked at the code for ages but I probably launched between five and seven attempts to buy the car and based on the amount of lag time that I was going to anticipate at the sale time I would launch them just a little bit before incrementally before the sale time and this was real successful so now there would be a number of BOTS and each one of those basically had a warhead that launched multiple attempts to buy the cars and so our success rate prior to making this fix during the competition was about he was getting about 50% after it we were back right on the money we were getting every car we wanted and it stayed that way through the duration of this program so how successful was the bot these are all guesses okay because I don't have any hard facts here but I know it was an operation for about 40 weeks and they were buying roughly five cars a day so there's about eight hundred cars I'm going to estimate were purchased with this if you figure the average wholesale cost of the cars they were purchasing was probably around $16,000 so in a 14 week period this bot purchased almost 13 million dollars worth of cars and that has a huge impact on a small dealer like this one so this is a great example of not accepting the web as it is not using browsers where everybody else would and doing something different and not being afraid to step outside of the box a little bit so what would I do differently today if I was going to do this first there were things that were done pretty well back then and things that I still do today I really like having very lightweight clients the lighter the better everything is easily updated because it was all online and it was easily distributed I can make changes on the server it would get distributed everywhere because basically they'd be sort of just these BOTS clients were essentially just web pages with some JavaScript and stuff going on one of the things that I really definitely would do if I were to do this over is I would build in some analytics and collect metrics so I would really want to know exactly what our success rate was I would want to know exactly how much these cars were purchased for it would be really great to also know how much they were sold for so I could actually show value that's something I really wish I had done the other thing I think that would have been nice if I were to do this over again is build in some some process that actually assists in the selection of which vehicles you want to purchase so in other words maybe what I would have done is I would have also had my BA look at Kelley Blue Book and figure out what the good wholesale prices are for cars and see look for discrepancies locate the ones that are that are under priced that would have been a really good thing to do the other thing that occurred to me actually within the last week is probably the only thing I really need to do here is make that Buy It Now button happen right and I could have done that simply by making the server act canna like a proxy so as the HTML is coming in with the grayed out button I could have just replaced it with a real button and send it off to the browser right that probably would have worked the problem there is that conceivably you could have bought cars before the purchase time and that may have been allowed but that's something you don't want to do for the same reason you don't want to buy cars that don't exist you don't want to you don't want to show your hand the website the target was a very traditional website it used HTML forms which are really easy for me to emulate or submit using just PHP and curl today you don't find that so often you find a lot of JavaScript you find a lot of Ajax there's a lot of JavaScript validation of form data before it's submitted it makes it a lot harder to do this kind of thing today so today the kind of approach that I take now is end up with a task queue which is basically a table in a database that keeps track of what needs to be done and there's a web interface into that so in this particular case my client would essentially be loading a task queue and that task queue would be fed to individual computers which I refer to as harvesters and they can exist anywhere they can be in the cloud they can be in a you know closet they can be in your office they can be anywhere in what I have them do now since there's so much more complexity in websites and so much more use of client-side scripting I do a lot of stuff in imacros anybody here use imacros it is the most amazing tool it's just an add-on for your browser that essentially lets you to create a macro for your browser that you can just play over and over again and what I do now is the harvesters will dynamically create that macro so you can get him to do some very specific things once I learn how to do that there was not a single website on the planet I could not manipulate it was like the gods handing me fire it's like here you're Mike you've been a good boy so that's what I do now and so I actually communicate through Firefox so it's very easy for me to emulate human activity now with BOTS so I would have them hit the sale server the difficulty there would be to get the timing down correctly but I think that could have been done and then the harvesters after they do their thing with the the sales server the target server they report back to the bot server and the queue is updated and that's how you can tell what the results were of what you did if you're interested in how that kind of stuff works go on YouTube and look up my DEFCON 17 talk because that's all about manipulating imacros and that way to to do screen scraper screen scrapers for very difficult to scrape sites or difficult to automate kind of sites so that's that's my talk thanks for all of you for coming thank you to the call for paper goons

Channel: HackersOnBoard

Views: 780,401

Rating: 4.6157784 out of 5

Keywords: 2013, defcon, 21, defcon21, t319, defcon 21 videos, def con 21, conference, hacking, leaning, learning, los angeles, Russia (Country), Soviet Union (Country), Botnet

Id: sgz5dutPF8M

Length: 26min 52sec (1612 seconds)

Published: Sat Nov 16 2013

Reddit Comments

[u/reefersaurusrex]

USA! USA!