Yesterday we incorrectly reported AWS was down. Our reporting fell short. So, we wanted to do two things: tell you what happened and explain the changes we are making to how we report cloud service outages to maintain your confidence in our reporting.
At first it looked like Amazon Web Services (AWS) was having issues yesterday (October 29) as outage reports started to spike on the monitoring website DownDetector.
According to the AWS service status page, all services were "operating normally," and the company told Tom’s Guide: "The only resource on the internet that provides accurate data on the availability of our services is the AWS Health Dashboard."
At the same time, we started to receive unconfirmed reports from users emailing about their own issues with “UnfillableCapacity” errors and a loss of Fire TV, seemingly caused by problems with Amazon's service.
What actually happened?
It comes down to two things: interdependence and the widespread impact of the very real Microsoft outage that hit yesterday. In particular, the Azure Front Door (AFD). An “inadvertent configuration change” caused Microsoft’s network and routing service to go down (DNS resolution failures).
Think of it like a massive, complex telephone switchboard that instantly connects a large chunk of apps and websites on the planet. AFD is the lead operator and accidentally flipped the wrong central switch.
While this was happening, user outage reports for AWS spiked too. My email inbox exploded with readers writing in, and X/Twitter jumped on the bandwagon. But many large companies and apps use a multi-cloud strategy — relying on both AWS and Azure for different services.
Once Azure flipped the wrong central switch, it disrupted services that also use AWS in its components, but the problem wasn’t AWS itself. “We are aware that an operational issue at another infrastructure provider may have impacted some customers' applications and networks,” AWS clarifies on its service health page.
Then came the perceived domino effect. AWS wasn't down, and Azure's outage did not impact AWS services. But false positive outage reports rose sharply alongside the Azure reports. Even Google Cloud's outage reports went up too. This is a regular occurrence when another cloud provider has downtime — effectively a flywheel that perpetuates itself for a while.
How we report on outages
It’s important that we make this clear. We serve you. Our readers, and nobody else. That is why we are led by a combination of three things: user outage reports on platforms like Down Detector, user-submitted emails documenting their own experiences, and company service status pages.
That being said, there’s an alteration we can make to our process of live reporting on possible cloud service outages. We must question first before declaring an answer in our headlines and coverage, because as you saw yesterday, interdependencies can cause confusion about what is down and what isn’t.
Only when we see something concrete on a cloud service status page, a company comment or verified evidence from our readers will we change our stance from saying “is something down?” to “something is down.”
We believe this is the fairest way to approach outages going forward. That being said, if you have any suggestions, questions or concerns, please do not hesitate to reach out to me in the comments section.
Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button!
More from Tom's Guide
- Microsoft issues urgent Windows 10 warning — and 84% of our readers are impacted
- Microsoft Teams will start snitching to your boss when you’re not in the office — and this update is coming in December
- Wi-Fi 8 just got its first ever demo — here's what makes it different
You must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.
