Understanding How Facebook Disappeared From the Internet

Tom Strickx and Celso Martinho, writing for the Cloudflare blog:

“Facebook can’t be down, can it?”, we thought, for a second.

Today at 1651 UTC, we opened an internal incident entitled “Facebook DNS lookup returning SERVFAIL” because we were worried that something was wrong with our DNS resolver 1.1.1.1.  But as we were about to post on our public status page we realized something else more serious was going on.

Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had “pulled the cables” from their data centers all at once and disconnected them from the Internet.

How’s that even possible?

DNS is deep dark stuff, and even at the pidgin level at which Daring Fireball operates, it terrifies me. Can’t even imagine how complicated it is at Facebook’s scale. What a fiasco.

★ Monday, 4 October 2021