What really caused the 2021 facebook outage (summary)

on october 4th 2021 facebook instagram and whatsapp disappeared from the internet for six hours this was one of the major internet outages in 2021 in this video i'll summarize the cause of the outage why did it take so long to restore let's get into it the internet can be thought of as network of networks connected and advertised by a protocol called bgp border gateway protocol this protocol defines proper routes to get from one network to another and allows for multiple path routing options and redundancy and also efficient shortest path routing networks constantly update the routes in case of a change to propose better routes or more redundant paths now that we know how the internet routes traffic what exactly happened that caused facebook to go dark around 8 50 a.m pacific standard time the facebook network stopped advertising its presence to their other routers on the internet this was due to a configuration change facebook made to their backbone router this means any ip address that belong to the facebook network cannot be routed to since there are no longer any paths available to get there the problem was first noticed through dns dns requests to facebook.com and whatsapp.com for that matter where timing out dns servers have ip addresses just like any other server and since facebook authoritative dns servers which are located on the facebook network could not be reached the dns resolvers such as 8888 which is google and cloudflare 1 1 1 couldn't route dns requests to them and as a result we get this 10 second default timeout in the dns request even if you did manage to get an ib address of facebook.com cached you won't be able to route to it as the paths have disappeared so why did the outage take this long to resolve my guess is internal tools that facebook used to make the configuration update we're locked behind and could not be accessed mainly because everyone's working remotely this might have made the job of reverting the configuration change even harder this is another reason why we didn't get any outage update even from facebook because there's no outlet to let us know that hey we're working on it because the facebook network is not reachable for those interested i'll be posting a new video detailing the incidents reading through cloudflare article reading through the facebook announcement and just to give my own thoughts stay tuned make sure to subscribe and i'm gonna see you in the next one goodbye

Channel: Hussein Nasser

Views: 35,323

Rating: 4.8776698 out of 5

Keywords: hussein nasser, backend engineering, facebook outage 2021, fb outage 2021, facebook, whatsapp, whatsapp outage 2021, instagram outage 2021

Id: PsLNQaY5SqE

Length: 3min 19sec (199 seconds)

Published: Tue Oct 05 2021