Facebook, WhatsApp, Instagram are down here’s what might’ve caused it (early report)

some breaking news facebook instagram and whatsapp are down and they usually go down this trio together because they all belong to the same company the parent company facebook they probably share the same infrastructure and some service probably one centralized single point of failure and uh i'm gonna give you a summary of what might be the cause obviously we know very little because this is there a very early situation and then we can go dive deep into what my preliminary analysis of the situation then we can we're going to call it a day i'm going to have to wait for the outage report from facebook so we can make more detailed assessment of the situation let's jump into it so it seems that this started around 9 a.m pst pacific time standard time and facebook and whatsapp and instagrams are not reachable you can feed it through the app first but if you even strive to go to the website facebook.com and whatsapp.com the dns never resolves to an ip address which is a very clear indication that the dns servers are not reachable or the dns couldn't resolve to an ip address we've seen a lot of outages start always with dns right microsoft big outage last year or earlier this year i mean had the same problem instagram on the other hand actually resolves to an ip address but what happens is eventually gets you a 500 error and this also might be a dns problem mainly because the back end of instagram could not resolve the dns to get to its back and feet that's again that's uh that's a summary so yeah it's boring dns i wish it was something as fascinating as the kinesis outage that was the most fascinating outage we learned so much from the amazon kinesis outage island so much it was a fascinating outage we need less boring outage this is a boring outage let's get into the details if you're interested all right i try to document this thing live as i just uh investigate the situation so what's up instagram and facebook are down when i tried to access whatsapp i got this error 502 bad gateway and this might be my fault the reason is because i had men in the middle proxy on my phone so i can intercept any requests right and that's that's a clear error bad gateway from men in the middle telling mew that you couldn't reach whatsapp.com if you did if you did try to go to whatsapp.com directly the dns wouldn't even resolve and wouldn't give you anything the same thing was for instagram instagram it was actually different instagram does resolve but it gives you like this 500 server error we're going to get into more details i was able to extract more useful details from the response header of instagram we're gonna get to that but uh going to facebook.com same same thing we couldn't resolve but that's why you get a bad gateway that's the man in the middle proxy that i installed giving me that error effectively so yeah i mentioned that instagram is the only back and i can tcp so i can establish the tcp connection i can establish tls 1.3 to the instagram reverse proxy whatever that thing is to the ip errors that that dns resolve to and we get a response haters that's an interesting thing look at this we got a beautiful beautiful so good look at this reverse proxy actually tells us that the proxy status this is fascinating it tells you the proxy status no server available so yeah so the reverse proxy clearly couldn't resolve a back and to serve and to kind of just send the request to so it just said die that's why that's just the the that's the idea i think that dns is the problem here for some reason instagram uses another dns provider right not different than facebook but that's just the idea i think this is all a shared pool of servers and all they they're all using uh facebook dns and that thing is down right let's continue let's continue what did we have what else i found okay this is a this is just what i found i found that there is a proxy status header there is a pro e s proxy error property there is a bunch of string i didn't want to copy it here or show it to you because i don't know if you know because sometimes facebook and instagram show you details about your machine about your ipads i don't want to show it to you guys like because i have no idea what this thing is but you can just do a curl dash v you can clearly see that thing right i couldn't decode it looks like a base64 but i couldn't decode it so we don't know what the error is but if i would guess is dns probably very odd again so i then though earlier i didn't know was dns so here i actually confirmed i tried to do ns lock-up on facebook.com can't find server whatsapp can't find server instagram actually we got our server we got this ip address from instagram and yeah so if i would guess again guys i think it's a it's a dns problem maybe right 99.999 percent it's a dns problem and that explains why why instagram couldn't reach its back end that also explains why you can reach facebook directly facebook reverse proxy and whatsapp reverse proxies right and and if this thing is down it's it's a single point of failure i know people don't like to call dns a single point failure because dns is distributed by design but in the in in the grand what do you call this a grand things grand scheme of things look at that is it's trending instagram facebook is down yeah in the grand scheme of things uh it is technically a single point a global single point of failure even that that because because you're relying on this thing that is dns yeah but but eventually dns will always talk to the rootwork to the root until it finds the authoritative and that if the authoritative is down you're done effectively right and that that makes it kind of a point of a single perfume i don't know guys uh what do you think what do you think might cause it is it more something more interesting than an actual um dns i i really i really hope so but let's wait and see the facebook report and uh it seems like uh this is this thing's still down let's take a look let's see if this thing is still down bingo facebook.com it's still down all right guys uh i'll update you if i find anything else talk to you later bye bye

Channel: Hussein Nasser

Views: 75,583

Rating: 4.8687 out of 5

Keywords: hussein nasser, backend engineering, facebook outage, facebook instagram outage, facebook outage 2021

Id: xdYhwW2-6_A

Length: 7min 21sec (441 seconds)

Published: Mon Oct 04 2021