Explaining the GitHub DDoS Attack

hey everybody John wagging on here with dev central and we're coming to you with another light board lesson video and today we're gonna talk about a recent DDoS attack that was just launched at the very popular web site github so you guys may have seen this in the news but I wanted to kind of break down exactly what happened in the attack and explain the methodology behind it and it also give you a little bit of maybe some advice on hey this is how you can mitigate this for yourself if this were to happen to you all right so last week github was the victim of the largest DDoS attack ever recorded in human history I guess we can say that so what happened was the attackers took advantage of this this caching system known as memcache D and so to understand the attack let me explain the way that that system works first and then I'll explain the attack all right so memcache D I'll just write that up here mem cache D memcache D is a caching system it's a distributed open source caching system then a lot of websites that use extensive database calls will implement because it allows information to be cached there and be presented to a user in a much quicker way than if you were to actually you know called the database and then get you know data back from the database and then present that to the user so basically what this is is you have a distributed network of servers so I'll just kind of draw like you know if you use servers here you know you can employ however many you need but these servers certainly have memory on them memory and and so whenever a whenever a user requests a website and a lot of social networking sites or sites that again have a lot of extensive database we'll employ this if a user will users out here and request your web server so I'll just say you know sends in a request to your to your web application let's say it's a web app then the web application the user may say hey I want the latest you know update on this picture or I want the latest whatever and in order to give the latest thing the truth is the data that resides there is probably held somewhere in some kind of database I'll draw a database back here as well so you've got you know a system of databases back here and you may have several of those as well so instead of the web server querying the database directly the the web server is going to say hey memcache D server do you have the data and if the memcache D server has the data it's going to respond and then the web application is going to present that to the user it's much much faster like I said then this then the web application of the web server coming down to a database making a database call doing a read and write just all this stuff this right here is much much slower so the nature of the way that this memcache D server system works is that the most recent data is held in these memcache D servers and so if there's any kind of older data that kind of thing it's gonna it's gonna hold the newer stuff you know instead of the older stuff but nonetheless it's going to fill up with tons of web server data and it's arbitrary data it's it could be it could be session data all kinds of different stuff can be stored here all right so that's kind of the nature of the background behind memcache D again user accesses your web application we're going to go to we're basically going to go to to the the memory like the caching system before we go back to the database that's the inherent you know process for how this thing works a couple of things though that are interesting about memcache D servers when these were set up they were set up with I'm gonna I'm gonna write a couple of attributes here I'm gonna put no off or no authentication at all they were they were never intended to be exposed to the public Internet and so the idea is that we're not going to introduce authentication to these because no unauthenticated person is ever necessarily going to access these you could argue the debate about you know did they need to add that or not but the but the point is no authentication mechanism was ever designed in the memcache D system all righty so they don't have that and they're never supposed to be exposed to the public Internet like I said a couple other little things is that one megabyte is the I'm gonna put limit per piece of memory stored on this server so any one piece of data as it were that's stored on these servers it's limited to one megabyte in size so anyway so whenever whenever someone asks for something anyone any one piece is going to be one megabyte or smaller okay another thing that you can do because there's no authentication is anyone can load data so I'm just gonna put load data on here on these things alrighty so there's no authentication you ought to tell it who you are and like any kind of caching system I guess that you would think about it stores data and it allows for a user to load data on to it alright so with all that in mind what can happen is you could have an attacker out here so I'm gonna put you know attacker bad guy out here attacker can can first of all scan the internet this is this is what happened to github the attacker can scan the internet and say hey Internet show me any kind of Internet public internet facing memcache the server system out there and and one of the recent reports that I just saw showed that there was in the neighborhood of 100,000 of these publicly facing memcache D servers out there in the internet so 100,000 not just of the servers but of these distributed systems like individual instances of this you know this whole thing going on so a hundred thousand options for you to go and tap in and because there's no authentication you can just get right into it you can start loading data onto these servers and certainly you can request data from those servers so as soon as you find these publicly facing memcache these servers what the attackers do is they say they send a get request so they'll say get you know I'm just gonna put data whatever and then they'll say the source IP I'll just say source IP is I'll put you know 1.2.3.4 just for example purposes alrighty well let's say that here's github down here github is doing its thing it's having a great day it's keeping all its customers happy it's not doing anything to anybody right and let's say github x' IP address is 1.2.3.4 alright what the attackers do is they say hey publicly facing memcache the server system I want to get data maybe data that I just loaded on there that I know is big which which again by the way because there's no authentication because these things are able to be manipulated because inherently they're supposed to be in a protected area this 1 megabyte limit can also be configured so you can say hey you know what allow 2 megabyte limits now or allow 3 or whatever you want to put so maybe an attacker gets in there changes the limit size on any one piece of data and then loads a ton of data in there and then sends a get request for that data all right so you can start to see how all this is going to build up well whenever it sends that get request the memcache D server sends a stream of UDP packets I'll just put I'll put you the P packets and they're going to be 1,400 1,400 bytes in size and it's just going to stream those packets to the recipient of the get request well if you spoof the source IP address the attacker does and you make the source IP address the IP address of your victim then this memcache D system is going to send all of those packets to the victim and so the victim again it's sitting there github having a great day doing its thing doing great doing great work and then all of a sudden it gets flooded with all these UDP packets from a memcache D server and they're like what's going on we didn't even ask for these and the attacker is like I know you didn't ask for them I asked for them on your behalf but you're getting flooded so what happens or what happened to the github or in the github instance is the the flood of data that came into github was 1.35 terabits per second traffic that just flooded this thing and then and that lasted for about eight minutes actually which is not a not a really long time lasts for about eight minutes the attackers kind of settled down a little bit and they came in a few minutes later they ramped it up was it quite 1.35 terabits per second of data flooding their site but they did kind of hit again and then they settled down so after about twenty minutes or so the whole thing was over which is a it's good news for github because it didn't last that long but it leaves us with some interesting points still to talk about one is that when you send a get request when an attacker sends a get request these memcache D servers they can send a very very small request so like this get data the the request size of that can be just literally just bytes in size but then the actual response these UDP packets is just going to start streaming megabytes and megabytes of data to the victim and so that's what that's the nature of what we call an amplification attack so the amplification attack is that you amplify the size of the request so the size of the request again maybe we'll say it's 200 bytes for example well then when the response comes out from these servers it may be you know a hundred megabytes plus in size so the response is an amplified version of the request so the attacker doesn't need that many resources frankly to realize a massively huge attack against a victim all right so that's so so that's kind of the nature of the amplification attack github did is whenever they started to detect this massive influx of this data or of all these you know packets coming in then they they rerouted I'll put a put down here DDoS scrubbing scrubbing Center alright so they have this DDoS scrubbing Center that they use and when they noticed that all this data was coming in that they were getting DDoS attacked then they routed all their traffic through the DDoS scrubbing Center the DDoS scrubbing Center by nature is going to take all of the data and it's gonna drop all the DDoS bad traffic and then it's going to allow the legitimate traffic to come on through into in this case github which by the way f5 has this exact capability we can we can detect the DDoS attack we can scrub it we've got cloud base scrubbers we've got a lot of different options for that so if you're ever interested in that you can check it out it's it's really cool technology but anyway once the once the scrubbing Center did its job I think that's part of why the attackers backed off and they said ok we're gonna be done with this because the data scrubbing Center did its job and then github you know came back online they didn't have to route the traffic through them through the DDoS scrubbing Center after a certain period of time because they knew that the attack was over so so that's that's the way that they mitigated and kind of got around this thing some organizations say hey we're just gonna have a massively huge pipe into our web application so even if we get ddosed then we can handle that and that's I would say that that's good to an extent but with the size of these DDoS attacks now it's gonna be very difficult to create a pipe that's big enough to handle all of these things so you probably want to employ some kind of scrubbing Center the other thing that I would note very quickly is that if you are an organization that utilizes this memcache D you know caching system then make sure that your servers are not publicly exposed to the Internet but because you may be part of the problem so so that's one thing we can do on the hey let's clean up the internet side of the house is that we can make sure these in memcache D servers are not internet exposed to the Internet if you're the customer side of this house and you're just trying to run your web application I would say you need to have a DDoS mitigation plan in place having a scrubbing Center as an option is it's probably a good idea depending on you know your business needs and all they have but it's probably something you gotta look into all right so that's the that's the basics of what happened to github 1.35 terabits per second we've never seen that before and good for github and their DDoS mitigation you know mechanisms they were able to to mitigate this one and there was not a huge hiccup but again the bad news is that there's still a hundred thousand of these memcache D systems out there exposed to the Internet so attackers could totally do this again either against github or some other unsuspecting victims so so be safe out there make sure you stay stay alert that DDoS attacks are real and they're they can they can be very that they can do a lot of damage so so thanks for watching this light board lesson video with us today if you like this you can subscribe to our YouTube channel by clicking on the DC ball right here and we will see you guys out there in the community you

Channel: F5 DevCentral

Views: 27,475

Rating: 4.9406781 out of 5

Keywords: f5, devcentral, memcached, github, ddos, amplification, attack, security, mitigation

Id: 4fRPcbgAcNA

Length: 14min 43sec (883 seconds)

Published: Tue Mar 06 2018