Facebook, Instagram and WhatsApp back online after massive outage — what we know

The Facebook and Facebook Messenger logos next to each other on a laptop screen and a smartphone screen.
(Image credit: viewimage/Shutterstock)

Update: Facebook, Instagram, WhatsApp, Messenger and Oculus VR are back online as of 6:10 p.m. ET.

The dust has settled after Facebook's six-plus-hours-long outage yesterday, October 4. As the social media giant's blue website went down, so did Instagram, WhatsApp, Messenger and Oculus VR. 

The company blames what it calls a "faulty configuration change." Essentially, the routers that chauffer people's web requests went offline. This had a rippling effect that brought down all facets of the entire company. A source told NBC News that employees couldn't even get into conference rooms as scanning ID cards all go through the same system. 

"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," said Santosh Janardhan, VP of infrastructure at Facebook via a blog post. "This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt."

Facebook's extended hiatus not only prevented aunts and uncles from sharing baby photos, but, according to a report by the New York Times, businesses in Turkey couldn't sell items nor could non-profits in Columbia use WhatsApp to communicate with victims of gender-based violence. It goes to show how one company has become so significantly ingrained in every part of human life. That's why there have been calls from antitrust watchers to break up the company. 

Not only that, a Facebook whistleblower, Frances Haugen, is set to testify in front of Congress today about the way the company has hid its own internal data about its harms. Not only that, but she claims that Facebook consistently chose profit, even if it meant pushing more caustic content. 

Below we've left an accounting of our updates from October 4 intact for future reference.

Facebook, Instagram and WhatsApp outage: latest updates

  • At exactly 11.39 a.m. ET, Facebook, Instagram, WhatsApp, Messenger and Oculus VR went down. The outage is now closing in on the seven-hour mark.
  • Facebook has not given a reason for the outage, but has tweeted that its working on getting the issue resolved. 
  • According to investigative journalist Brian Krebs, when speaking to Doug Madory of Kentik, a network observability company, Facebook's DNS records were withdrawn from global routing tables. 
  • According to Philip Crowther of the Associated Press, it's "mayhem" over at Facebook. Internal systems aren't working. Employees are communicating via text. 
  • According to the New York Times, Facebook employees' badges aren't even working, meaning they can't enter the building. 
  • As of 6:10 p.m. ET, Facebook, Instagram and WhatsApp are back online. We're still waiting on an official statement from Facebook regarding the outage.

While the Twitter accounts for Facebook, Instagram and WhatsApp were slow to update users, all three services have since put out short statements. Each is a variation of acknowledging the issue and telling users that work is being done to address the issue. Still, there's official release on when this outage will end or why the outage has occurred. 

Of course, a flurry of silly memes poking fun at Facebook and its suite of services have begun cropping up, with many lauding Twitter for being active throughout all of this. Twitter itself is now in on the joke. 

Possible outage cause

As to why the outage has occurred, that has yet to be determined. But according to investigative journalist Brian Krebs, when speaking to Doug Madory of Kentik, a network observability company, Facebook's DNS records were withdrawn from global routing tables. 

DNS stands for Domain Name System. It's how the name of a website, such as tomsguide.com, is translated into a raw IP address. When the DNS isn't working, web browsers can't find the website being called. 

It also seems that the BGP, or Border Gateway Protocol, routes have been pulled from the internet. Cloudflare describes BGP as the postal service for the internet. When someone wants to access data over the internet, it's BGP that tries to find the fastest route possible. Without BGP routes, there's no way for data to access Facebook, Instagram, WhatsApp, etc. Not only that, Facebook itself can't communicate within Facebook.

According to ArsTechnica, a user on Reddit who claims to be a Facebook employee, posits that network engineers may have been pushing a configuration change that accidentally locked them out. In this case, local data center technicians with local physical access to the routers are the only ones able to fix this. Per this Reddit user, the outage is not because of a malicious attack. 

"The ongoing outage of WhatsApp, Instagram and Facebook (including Facebook Messenger and Facebook Workplace) highlights that global outages are one of the major downsides of a centralized system," said Matthew Hodgson, CEO of Element, technical co-founder of the Matrix open standard, in a statement to Tom's Guide. Element is a decentralized collaboration and messaging platform and Matrix is an open source API for decentralized communication with end-to-end encryption. 

"Centralized apps mean that all the eggs are in one basket. When that basket breaks, all the eggs get smashed. We saw the same last week when Slack went down."

Hodgson argues that a decentralized system is ultimately more reliable as there's no single point of failure. 

Today's outage is the second longest in Facebook history. The longest still goes to the one that occurred on March 13, 2019, which lasted nearly 12 hours

Other trouble at Facebook

While there's no official statement as to why the outage has occurred, but for Facebook, this news comes at the heels of multiple damning reports. It's likely why #DeleteFacebook is also trending on Twitter.

Not only has the supposed DNS delisting thrown Facebook off the internet, it seems that internal systems across the company are down. According to Philip Crowther of the Associated Press, in speaking to a source at Facebook, "it's mayhem over here, all internal systems are down too." Employees are being forced to communicate via text message and Outlook email systems. Not only that, employee badges aren't even working, meaning many can't even enter the building. 

60 Minutes recently published a report in which a former Facebook employee, Frances Haugen, blew the whistle on internal practices at the company which encouraged the spread of anger-inducing content as a ways to keep up engagement. This, of course, has led to much criticism being lobbed at the company as putting profits over civic health. Facebook, in internal memos, has denied its platform as being used as a tool which led to the Capitol insurrection on January 6 of this year. 

The latest news on Facebook comes after the Wall Street Journal reported last month about the company squashing its own research about Instagram and the affect it has on younger users. It's mixed up in now cancelled plans for Facebook to launch Instagram for kids, which caught the alarm of multiple officials and led to lawsuits by attorneys general from 44 states. 

TOPICS
Imad Khan

Imad is currently Senior Google and Internet Culture reporter for CNET, but until recently was News Editor at Tom's Guide. Hailing from Texas, Imad started his journalism career in 2013 and has amassed bylines with the New York Times, the Washington Post, ESPN, Wired and Men's Health Magazine, among others. Outside of work, you can find him sitting blankly in front of a Word document trying desperately to write the first pages of a new book.

  • WillSmartHome
    I spent 32 years in IT with major corporations, clearly, the Facebook infrastructure is seriously flawed. For a company completely dependent on Internet presence as their sole business model they should be on a virtually failsafe infrastructure. If Amazon never goes down then Facebook should be on AWS infrastructure but they thought they knew better. Obviously, the organization is also seriously flawed but of course, we already knew that.
    Reply
  • USAFRet
    WillSmartHome said:
    If Amazon never goes down then...
    From today, Oct 4
    https://downdetector.com/status/amazon/
    Reply
  • WillSmartHome
    Well you showed me something! But was Amazon down totally or just a higher number of reported problems? I seem to remember one of the hacker groups throwing in the towel a few years ago because they simply could not overwhelm Amazon.

    If I were the CIO at Facebook I would have minimum 3 redundant servers farms at each site and then at least 6 worldwide IT sites with the ability to host all the traffic at any one site perhaps with performance hits but definitely not an outage. Facebook has more than enough money to build massive redundancy and extreme availability with failover almost instantaneous to another IT site. In today's world highly available and massively parallel application servers are known technologies as are the data replication and the failover technologies.

    They have just not decided that they want to pay for true high availability and redundancy and all of the testing required to continuously validate the architecture and resources. Their objectives are very straight forward maximum Availability and Security. For that, you hire the best and you pay for the required infrastructure.
    Reply
  • rgd1101
    I read the issue was the DNS got mess up. nothing sure how many servers farm would help.
    Reply
  • WillSmartHome
    rgd1101 said:
    I read the issue was the DNS got mess up. nothing sure how many servers farm would help.
    There are companies that specialize in highly available and failover DNS infrastructure. Selecting the best of these is part of your High Availability design.
    Reply
  • rgd1101
    sure hardware redundancy is good. but that not going to help with bad setup/code. Much like people thing raid would fix everything

    read this.
    https://blog.cloudflare.com/october-2021-facebook-outage/
    Reply