Why was Facebook down for five hours?

last week all of facebook services were offline for about five hours and i thought it'd be interesting to discuss it we'll start with the idea that every computer on the internet has a numerical ip address that it uses to communicate and so one of the first things that has to happen when you type in something like www.facebook.com is that your computer has to look up the ip address for facebook and the way that works is using dns the domain name system but that just begs the question how do you find the ip address of the dns servers well ultimately it all starts with the root servers there's a well-known public list of 13 ip addresses for the root dns servers and the list is maintained by the internet assigned numbers authority which everyone just kind of trusts but the actual servers are operated by different corporations uh educational institutions government and and military agencies uh and and various nonprofits so really no single entity or interest controls the internet and all of these root servers provide the same information and each server has a well-known published ip address and those are all listed here so for example this root server has the ip address of 198.41.0.4 and an ip address is really just a 32-bit number but it's always written out as four 8-bit numbers separated by dots like that but anyway everyone agrees that these 13 ip addresses are the authoritative dns root servers so we can send a request any of them to try to figure out the ip address for something so for example here i'll do a dns lookup from one of the root servers so 198.41.0.4 and that's this root server here so we're going to ask that root server what it knows about www.facebook.com and so we do that we get back a whole bunch of information but if we look at what we actually got it says we queried for one thing and we actually got zero answers so this is not actually going to answer our request but it did give us a bunch of other information so it's saying here our question was www.facebook.com we're looking for an internet address that's what we're that's what we're after it doesn't give us an answer but it does give us some other authority so it says that all of these different servers are authoritative for com and so what it's saying is the the root server is saying okay you're looking for www.facebook.com well i don't know anything about that but i do know about com this top level domain and if you want to know anything about the com top level domain well then you can talk to any of these 13 servers which coincidentally are managed by the same organizations that operate the root servers in any event it says we can ask any of these generic top top-level domain servers for more information about the com domain and then it also provides us this additional section which has some glue records that gives us the ip addresses for each of these servers that are authoritative for the top level com domain and i'll note that in all of these cases we have this ip address that we've been talking about as well as an ipv6 address and the difference is that an ip address is a 32 bit number which is written out as four eight bit numbers separated by dots whereas a ipv6 address is 128-bit number that's written out in a hexadecimal format and the internet is slowly moving towards adopting ipv6 more broadly because it supports a much bigger address space and so we could have just as easily used this ipv6 address and asked the same question so we could ask that ipv6 address what it knows about www.facebook.com and we would get the same answer so this is the exact same answer it says oh well for com you've got to talk to one of these servers but anyway for the rest of the video i'm going to be focused on ipv4 because the numbers are sort of easier to make sense of so anyway the next step is once we've got this answer saying that okay for com you need to talk to one of these servers well then we can ask one of those servers so 192.12.94.30 was one of the servers that is authoritative for com we can ask it what it knows about www.facebook.com so if we run that we get another answer here where it says okay we queried for one thing we got zero answers so we didn't still didn't quite get the answer for who www.facebook.com is but what this is now saying is that you need to talk to a different server that is authoritative for facebook.com and in that case it lists these four name servers that are authoritative for facebook.com and it gives us the glue records here that give us the actual ip addresses so now if we kind of take the next step and ask one of these servers so 129.134.30.12 for example if we ask that server what www.facebook.com is here we get a result where we are getting an actual answer says we got one answer so our question was what is www.facebook.com we're looking for an internet address the answer is that www.facebook.com is actually an alias and the canonical name for www.facebook.com is starmini.c10r.facebook.com so we still don't actually have the ip address of www.facebook.com we're kind of going further down the rabbit hole so we are actually need to find the ip address for this name here starmini.c10r.facebook.com well we already know it's facebook.com so we'll ask the server that is authoritative for facebook.com which is the server we just asked here to see what it knows about this star mini thing and when we make that request would you note we don't get an answer what we see is we see another authority section we see okay you're asking for something within the c10r.facebook.com subdomain well for that you've got to go talk to one of these name servers so we'll go ahead and ask that same question but instead of using one of the facebook.com name servers we'll use one of these c10r.facebook.com servers so that's this one here 185 89 218 11. and if we make that request we actually get an answer so we get an answer again so starmini.c10r.facebook.com we're looking for an address and we actually get an address and the address is 157.240.22.35 and so that is in fact at least for me and at least right now that is the ip address that i should talk to if i want to talk to facebook.com but in order to figure out what that address was we had to start with the root servers and then they were able to tell us about the top level domain servers which are authoritativefor.com and then those top-level domain servers are then able to tell us about which servers are authoritative for facebook.com and then it was those servers that are actually able to tell us about www.facebook.com but then they just said that it's an alias for starmini.c10r.facebook.com and that there's actually different dns servers that are authoritative for that c10r.facebook.com domain and so it's finally those servers that were able to tell us the ip address of facebook's web server now for each of these zones there are multiple addresses and of course that's for redundancy and actually each of those addresses is likely served by multiple servers in different locations but still we need at least one address from each of these four zones to be reachable and operating in order for this whole process to work though i should say that in reality your computer probably doesn't actually go through this whole process so on my computer here if i'm looking at my network settings if i go look at dns it's got some dns servers configured and these are actually caching name servers that are run by my isp so for example 75.75.75.75 is run by comcast which happens to be my internet service provider so if i just do a lookup for www.facebook.com and i don't specify a server it'll use that comcast caching name server that my computers is configured to use and you can see that's what it used here 75 to 75.75.75 and it's sending the query and it's getting back two answers and the query is here www.facebook.com we're looking for an internet address and it's getting the two answers it's saying that facebook.com is actually an alias and the canonical name is starmini.c10r.facebook.com that's the same as we saw before and then it's saying that the star mini it's got an internet address of 157.240.22.35 so we're getting the same answer as we got before but instead of going through that whole exercise of going to the root server and working our way down we can send a single request to my isp's caching name server and and you know basically get the same answer right away but of course this caching name server still has to go through that whole process but then it can cache the answer so if someone else wants to know what facebook's ip address is it can just answer directly which is actually probably what happened in this case and actually the authoritative name servers say how long each result can be cached so if we scroll up here we can see that the the root server said that you know the fact that these name servers are authoritative for dot com can be cached for 172 800 seconds which is uh 48 hours and then you know the ip addresses for those can also be cash for 48 hours and then the top level domain servers when we asked about facebook it said that facebook's name servers can also be cached for 48 hours but then facebook's name server says that the www.facebook.com you know that alias to star mini can only be cached for 3 600 seconds or one hour and then the mapping uh down here of that you know starmini.c10r.facebook.com to the actual ip address can only be cached for 60 seconds and you know these intervals here are something that facebook has set up so there's you know sort of a trade-off between letting it be cached for a long time versus being able to change this address relatively quickly to handle failures or changing load or whatever so facebook has that set pretty aggressively low at 60 seconds but anyway what this tells me is that even though this address i'm getting from my isp it might have been cached my isp would have had to have gone back to facebook's authoritative name server sometime in the last 60 seconds to to get the updated address in fact i can see that because this expires in 33 seconds i know my isp actually got this information would have been 27 seconds before i requested it from them so even with caches it's really important that at least one root server top level domain server at least one of facebook's name servers and apparently also a subdomain server are all operational in order to get the ip address for facebook's website so if somehow all four facebook name servers were unreachable no one would be able to get the ip address for any other facebook service and of course that's what happened last week all four of facebook's name server addresses were inaccessible so what does that mean that they're inaccessible you know i mean let's say we've gone through this process of talking to root servers and top level domain servers and now we need to talk to one of facebook's name servers we've got an ip address for it 129 134 30.12 let's say how does data get from my computer to that ip address and really for that matter how does data get from my computer to any ip address well my computer of course has an ip address of its own in this case it's 172.16 so i mean it sounds kind of silly but at least at a minimum my computer is going to know how to get to this address 172.16.0.223 because that's itself but it's also got this subnet mask 255.255.255.0 and if you convert that to binary you get 24 ones followed by eight zeros and the meaning of that is that it says that the first 24 bits of this ip address describe the local network that i'm connected to and the last eight bits so in this case this 223 describe a single device on that network so if you change any of the last 8 bits of this ip address you just get another ip address that's on my local wi-fi network but for any other address that i might want to find that doesn't match that there's this router configured that's 172.16.0.1 and of course the first 24 bits of that match my local ip address so we can tell that the router is on the local wi-fi network here which is good because we don't know how to get anywhere else just yet but now that we've got this router configured that's going to be our default gateway to get to any other ip addresses on the internet and another way to look at this is to look at my computer's routing table and this shows routes to a bunch of different destinations and we can see we've got 172.16.0.223 and that's my computer's ip address and it says all 32 bits have to match so my computer knows how to get to itself but here's 172.16 24. so any ip where the first 24 bits match is 172.16.0.0 to the same wi-fi network that i'm on but for most other addresses there's this default route and the next top is the gateway of 172.16.0.1 and that 172.16.0.1 belongs to my router on my home network that sits between the wireless access point and the cable modem that connects to my isp and you can see that router's got two interfaces and and this one here is the 172.16.0.1 and this other one connects over to the cable modem and i can actually connect to my router and poke around if i secure shell to 172.16.0.1 now if we look at the interfaces you can see that ethernet 1 is assigned this address of 172.16.0.1 and then ethernet 0 has this 24.6.218.102 address which is assigned by my isp since ethernet 0 is connected to my cable modem now if we look at the routing table it's pretty simple we see this 172.16.0 24 network is connected to ethernet one and then the 24.6.216.21 network is directly connected to ethernet 0. and this slash 21 means that the first 21 bits of this are the network and then the 11 remaining bits are used for identifying me and about 2 000 other homes in my neighborhood and so my address of 218.102 falls within that 216.0621 range and then this is the default route here so it says that for any other address that matches if we compare zero bits so that's be any address send it out my cable modem and the next hop is 24.6.216.1 and that would be a router at my isp that i'm directly connected to so this is all pretty straightforward every device on my network has a default route that points traffic in one direction towards a bigger network but once you get to an internet service provider's network you can't do that anymore the isp's routers don't have a default route they need to know where everything is so to see what that looks like i'm connected here to a router on att's network that i happen to have access to and if we look at the routing table on this router first thing i'll point out is that this routing table has information for 845 478 destinations and those destinations each destination refers to an address prefix that can cover a big block of addresses but but all of these 845 000 destinations together cover the entire internet you can see there's no default route here and it just starts at 1.0.0.0 and and goes from there and you know some blocks are bigger than others but if we just keep scrolling through here it just goes on and on and has blocks for every active ip address on the internet and you know this happens to be at t's view of the world but any large service provider is going to have a similar looking routing table in their routers and likewise facebook operates a sophisticated global network so they aren't just going to have a default route pointing somewhere and i don't really know any details about facebook's network but i mocked up five routers configured you know close enough to how it might work just to kind of show you and i you know i just made up some ip addresses and stuff but we can poke around and look at stuff so if we're on the menlo park router here and we just look at the routing table you'll see there are you know 40 destinations and i just made up some ip addresses but the important point is there's no default route and i'm pretty confident that facebook wouldn't be using a default route so they'd have a full routing table just like we saw on that att router but anyhow since this is supposed to be a mock-up of facebook's global network let's add in one of their dns servers so a.n.s.facebook.com has an address 129.134.30 so i just arbitrarily stuck that in montreal but they almost certainly have multiple dns servers responding to that address in multiple locations so i'll add a second server with the same ip address in amsterdam as well and that'll work perfectly fine for example if we're looking at this routing table on the menlo park router there's a route here in the routing table 129.134.30.0 24 which covers that dns server's address and there are two paths here and you can see one path goes to montreal and the other path goes to amsterdam and the first one here is selected as the active path to the star means and the second one is just there serving as a backup but if we move over to the london router and look at the routing table there you can see the same 129.134.30 24 route and it's also got two paths one going to montreal one going to amsterdam but the london router selected the amsterdam one as the as the preferred route which which makes sense but again it's got that backup route available as well so that's a mock-up of facebook's network but you know i also created a mock-up of my isps network and in this case this one's a lot simpler i just have a router in san jose near me and then another router just in london for example but you can imagine that this isp might also have routers all over the world but i'm a customer of this isp so my house is connected to their router here in san jose so i've mocked that up if we look at the routing table on this router and specifically look for that address 24.6.218.102 you can see that network is directly connected in reality this would be through the cable modem infrastructure but i've just got this mocked up and then the isp of course is going to ensure that the rest of the network can get to me as well so if we go to the isp's london router there should be a route there as well and we can see that same route to that same slash 21 network for my neighborhood and it's advertised from the san jose router and the next top is 1015. which is the san jose router via interface em2 which is that link on the london router going to san jose so so that all checks out but so far none of the routers on my isps network know how to get to facebook's name server if we go back to the isp's san jose router and look for a route going to facebook's name server 129.134.30.12 we get nothing and remember on these big networks there's no default route so any data trying to get to this address would just be drop because we we've got no route there likewise if we go back to facebook's network so i'll jump over to that menlo park router none of the routers on facebook's network are going to have a route back to my cable modem and of course i wouldn't expect facebook to have a default route either so we've got these two different networks but they aren't connected to each other in what you might call an internet but of course both of these companies facebook and the isp would benefit from connecting the two networks together facebook wants access to the isp's customers and the isp's customers want access to facebook so we can add a connection between the two networks um but you know facebook isn't exactly a regular customer of the isp and the isp isn't exactly a customer of facebook so it's really more of a peer relationship and so this type of connection between networks is typically referred to as peering and it's even common that neither side pays the other because it's just a mutually beneficial thing to do but anyway setting up peering uh can be pretty straightforward you know first you need a physical connection between one of facebook's routers and one of the isp's routers so you know menlo park and san jose are pretty close to each other so might be easy enough to get a fiber optic connection between those two facilities although it's actually quite common for different companies to put their equipment in the same data center so maybe both of these london routers are actually in the same data center and you can actually just run a wire across the room to connect them so anyway i've got these connections set up between san jose and menlo park and configured with these ip addresses so for example if we look at the interfaces on the menlo park router we've got this em3 interface configured with the address 1016.2 and if i ping 10.0.16.1 i'm getting replies so the link is up and the facebook router and the isp router can send traffic back and forth but what's missing is that facebook doesn't know what traffic it should send to the isp and the isp doesn't know what traffic it should send to facebook so what we need to do is we need to exchange routing information across this link that we just created and the way that's done is using the border gateway protocol or bgp and so with bgp what we do is we set up a tcp connection between the routers to exchange routing information and i keep track of who owns what routes and what networks they've been advertised through every network that's part of the internet has a number assigned to it it's called an autonomous system number for example 32934 is assigned to facebook and so i'll use that in my demo and then of course i've also got the isp network and so that needs its own autonomous system number and uh as number 7018 happens to be assigned to at t so i'll just use that for my demo and all the routers on a particular network are going to be configured with that autonomous system number so on the isp side if we go to say the san jose router and look at the configuration i've got it configured for autonomous system 7018 already and then same thing on facebook if we were to hop over to one of the facebook routers say the facebook router in new york you can see i've got the autonomous system configured for 32934 which is facebook's as number and i've already got that configured on all the routers so to set up the peering between facebook and this isp i'll start on the isp side so to set up that peering i'll configure a new bgp neighbor and that neighbor's going to have an address of 10.0.16.2 which is the other side of that link to facebook and then i'll tell it that the pure autonomous system is 32934 and that's actually it so i'll commit that change and exit config mode and so now if we look at the bgp neighbors there's a neighbor configured for 100 16 2 and it's active which means that it's actively trying to establish a connection but it's just going to stay in this state now until we configure the other side so let's go over to the facebook router now and it's basically the same thing like i said i've already got all the facebook routers configured to be in autonomous system 32934 so we just need to set up the peering on this side so i'll add a new bgp neighbor with a neighbor address of 10.0.16.1 which is the isp side of the link and then i'll tell it that the pure autonomous system is 7018. so i'll commit that change if we look at the bgp neighbors here's our new neighbor 1016.1 and it's showing as active so it's actively attempting to establish that connection if we keep checking it it should eventually become established and start exchanging routes and there it goes so now it's exchanging routes so we're we've got 12 active routes 12 received routes 12 accepted routes so we're getting 12 routes from the isp and if we go back over to the isp and look at the bgp neighbors again here we now see that this neighbor is established it's been up for 23 seconds and it's receiving 22 routes from facebook so now from the isp if we look through the routing table we should see those facebook routes so here from san jose here's a route to 129.134.30 24 which includes facebook's name server and we see we're getting a bgp route and it shows that the as path is three two nine three four so it's telling us that this route has come from facebook's autonomous system and then if this isp were to be peered with another isp and send this route on to them then this isp would prepend its as number and this path would keep a record of how many networks the the route has been advertised through and it's also used to detect loops because if you receive a route that includes your own as a number in the path then you know well it must be there must be a loop or something and you can discard that route now if we jump up to one of facebook's routers let's say in montreal let's see if there's a route here for my home address which was 24.6.218.102 and there we go so now facebook knows about my home address and it says to send traffic from montreal towards menlo park and if we go look at menlo park's routing table for 24.6.218.102 it says traffic should be sent towards 10.0.16.1 which is the isp router in san jose now it's pretty common for big networks like this to appear in multiple locations and i'm sure that facebook peers with big isps in probably hundreds of cities so i can set that up the same way in london we've got a link here between these two london routers i've already configured that and so on the isp side if we go to the isp router in london it's configured the same way we'll set up a neighbor address of 10.0.13.2 and we'll set up the pure autonomous system from the isps side looking at facebook to be facebook's as number of 32934 so i'll commit that and there's our noob here active and then on the facebook side this is a facebook router in london the bgp neighbor is going to be 10.0.13.1 and the pure as is 7018. so commit that and we've got that neighbor now and we'll just give it a minute here and it should uh become established and there it is so that's established now and so now with both of those peering set up if we go back to san jose and look at routes to 129.134.30.12 facebook name server you can see we've got two paths here one going out interface em2 towards menlo park and the other going out interface em-1 towards london and of course it prefers the one going towards menlo park if we were to jump over to london and look at the same route we see the same two paths and of course in this case the london router prefers sending it to facebook's london router so that's a super high level overview of how large networks like facebook and at t connect together to form the internet so let's take a look at what routing information at t really has about facebook so now i'm on that router that i've got access to that's uh actually connected to at t's live network and you know as we saw before it's got about 850 000 routes but we can look for routes that match a particular autonomous system path so we can say show us routes that have been advertised through any network any number of times but that ultimately originated at as32934 and that's of course facebook's autonomous system number so this will show us every route that facebook is advertising that's eventually making its way to this router somehow so here's a destination this 3113-24 and it's got a whole bunch of different paths to that destination and here's some more routes but we could we could simplify this a bit i'll modify that same command to just give us active pass so we don't have to look at every path for every route and so here are all of facebook's ip blocks that they're advertising and if we keep scrolling down here we should be able to find that 129 134.30 route and here it is 129.134.30 24. and of course you know we'd expect to see this route from facebook otherwise we wouldn't know how to get to their name server and so you can see all these routes here and i actually happened to take a look at this list of routes during facebook's outage last week as well and i put a list into this spreadsheet here so this column here is all the routes that facebook was advertising during the outage and then this column over here is all the routes that they're advertising right now and so if we scroll down you can see there's a handful of routes that they were not advertising during the outage and critically that included routes for these networks here that include the a name server the b name server the cname server and the dname server so this is pretty remarkable you know facebook has got four name servers at four different addresses and presumably each address is hosted in dozens or maybe even 100 or more different locations and facebook undoubtedly peers with hundreds of other networks and somehow simultaneously all four routes were withdrawn from every location to every pier and you know this sort of thing just isn't supposed to happen so what did happen well facebook posted an explanation and they talk about dns and bgp and they say that you know to ensure reliable operation our dns servers disable bgp advertisements if they can't speak to their data centers and i wouldn't think that would normally be a problem so let's hop back to my simulated facebook network here and remember we've got two dns servers with address 129.134.30.12. is in montreal and one is in amsterdam and if we look at the route to that address on any of the other routers say in new york that route has two different paths one to montreal one to amsterdam so one to each server so if one of those servers decided it was having problems and disabled bgp advertisements about itself as facebook says that they do then yeah the other one's still there and from what i can tell facebook has about 100 or so appearing locations and you know i wouldn't be surprised if they had dns servers in each of those locations so that might mean something like 100 different routers disabling route advertisements for their dns servers almost simultaneously but that seems to be what they say happened now why that happened they're not all that clear about so they say a command was issued which unintentionally took down all the connections in our backbone network and i don't know that's not a very satisfying explanation but unless facebook wants to tell us more this is what we've got the entire network was down because they unintentionally took down the entire network and you know beyond all this talk of dns and bgp that's that's really what we're left with and the rest of this is an interesting read you know they go on to talk a bit about the challenges they had restoring service and that may actually be the bigger story here you know mistakes happen but there's a big difference between a five-minute outage and a five-hour outage and it sounds like a lot of their time was spent fighting their own security protocols to get the right engineers into their own data centers and i'm sure they're not going to tell us the details about that but anyway i thought this was a good excuse to explain a little bit about dns and bgp so hopefully you found that interesting if you want to learn more about networking actually the very first dozen or so videos i made on this channel seven years ago who are actually all about networking so maybe check them out and of course i want to thank all my patrons who make it possible for me to randomly drop everything and make a video about dns and bgp for no reason so thank you

Channel: Ben Eater

Views: 135,189

Rating: 4.9670711 out of 5

Keywords:

Id: -wMU8vmfaYo

Length: 30min 35sec (1835 seconds)

Published: Tue Oct 12 2021

Reddit Comments

[u/Isvara]

I'm fairly confident that Facebook wouldn't be using a default route

At the edge, yes, but most of Facebook will be using (multiple) default routes.

It's common that neither side pays the other, because it's mutually beneficial thing to do

Unless you're Comcast.

[u/soks86]

Awesome video. Never wanted to take the time to learn BGP but I'm glad to know more about it now.

[u/DragonCalypso]

As programmers we sometimes focus too much on the technical details but not on the sort of organisational failures that can allow this to happen.