What You Need to Know About the October 4th Facebook Outage…



ICS Data | Facebook Outage


What You Need To Know About The October 4th Facebook Outage…

October 7, 2021 | by Jacob Acker


A few days ago, there was an interesting series of events that caused a Facebook outage. According to Facebook Engineering, “their teams learned that configuration changes on the backbone routers that coordinate network traffic between their data centers caused issues that interrupted this communication. This disruption to network traffic has a cascading effect on the way their data centers communicate, bringing their services to a halt.”

So, if you noticed that your Facebook and/or Instagram page was down on Monday, that’s the explanation as to why. In the same article, they provide an apology to “all of the people and businesses around the world that rely on them.” It’s a shame to see communications errors, but as we realize, they sometimes happen to even the largest of giants.

Let’s dive a little deeper into what exactly happened…

THE FACEBOOK OUTAGE EXPLAINED

Initially, based on this Cloudflare article, the problem began with a configuration change that affected the entire internal backbone of Facebook. That cascaded into Facebook and other properties disappearing and staff internal to Facebook having difficulty getting services going again.

Continuing, they mention that BGP (Border Gateway Protocol) is a mechanism to exchange routing information between autonomous systems (AS) on the internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network to the final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work. The Internet is literally a network of networks, and it’s bound together by BGP.

Providing a long story, short – Facebook stopped announcing their DNS prefixes, meaning that their service was unavailable.


DUE TO THE FACEBOOK OUTAGE, FACEBOOK.COM WAS FOR SALE

As baffling as it seems, this did actually happen. Stacked Marketer mentions, “it could have been a deal of a century.”

Because Facebook-owned properties were offline, someone performed some DNS magic on Facebook.com causing it to appear “for sale” on many registrar websites. Twitter’s founder, Jack Dorsey, was one of the first to offer to buy the domain (see image to the right).


ICS Data | Jack Dorsey's Twitter Post


WHAT DOES THIS ALL ENTAIL?


It’s true, even the biggest tech companies in the world can have failures that cause catastrophe. That’s why it’s important to understand that live [hot] changes can affect outcomes at any stage of your technological process. Meaning, it’s better to test [stage] processes before doing anything to affect internal measures.

This Facebook outage cost Mark Zuckerberg billions of dollars. Now, we’re not saying that it could cost you that much, but you would lose time, money, and possible assets in the same situation.

Let’s talk today and see what other technical adventures we can help with.