The Friday the 13th Tech Massacre

How the safety system became a failure.

On Friday the 13th (you can’t make this up), while validating the weekly maintenance window for its disaster recovery tenants, Workday’s DR services “unexpectedly connected to production databases.” This left customer data in an inconsistent state, causing transaction errors and failed jobs.

What followed was 11,500 organizations, including Target, Walmart, Charles Schwab, and Nasdaq, whose employees couldn’t clock in, view pay, request time off, or complete benefits enrollment. Workday tested its disaster recovery system, which did what every cyber hacker dreams of doing: brought down the entire SAAS program for all of its customers.

As the CEO of a technology company who pushes his clients to test their backups and disaster recovery systems regularly, I am fully aware of the irony. Companies should test their DR systems. Making sure they can’t connect to the production systems should probably be high on the list.

If the Workday outage were the only problem last week, it would be significant, but far short of a massacre. However, on Thursday, the 12 of November, Microsoft 365 went down hard. Microsoft’s preliminary finding is that “a failure in the systems responsible for establishing secure connections to the service” occurred when “traffic shifts during standard network maintenance and upgrades exposed a latent bug.”1  The outage lasted 24 hours, with some users reporting issues for up to 36 hours.

Not to be outdone, on Friday, the FAA issued a nationwide ground stop on all departures, with over 3,500 aircraft already airborne.2  Lifted four hours and twenty-one minutes later, this was the longest grounding in twenty-five years, and second only to 9/11.3  More than 4,700 flights were canceled and 10,000 delayed, with at least 827 diversions.4

The temptation here is to link all three events, or create a cyber-hack conspiracy. The reality is less sexy, but actually more important. The issues that occurred were structural, and probably not in the way you think.

The first one to examine is the FAA. Many want to pin this on its antiquated technology system. The older system was responsible, but indirectly. First, contrary to the original reporting, the radar never went fully down. Per the FAA, “inconsistency in the data from the radars,”5  and in communications with aircraft. The newly installed technology overloaded the older components. The radar didn’t go dark; it started giving bad data, which is worse.

A system that fails outright can be worked around. A system that fails quietly, by producing data the controller can no longer trust, has no workaround. This is why the FAA grounded all flights immediately after the discovery. Microsoft’s Exchange servers didn’t die; they slowed enough to fail a health probe. Workday didn’t outright go down; it produced transaction errors from inconsistent data.

You may be seeing the patterns. The systems put into place to prevent outages actually caused them. Microsoft has software that performs health checks on the servers. If a server stops answering, the system pulls it out of rotation and sends its traffic elsewhere. The Microsoft system pulled servers out of rotation, which then caused the servers still running to slow down; the health check then pulled them out of service, creating a death spiral. Bringing servers back online didn’t fix the problem. They immediately took on large amounts of traffic, and the Microsoft health check system would drop them again.

Microsoft had to switch to a validated alternative health check system to bring the servers back online. The system meant to keep their servers healthy ended up bringing the whole system down. This happened earlier to AWS in October 2025: Network Load Balancer health checks kept failing for instances whose network state hadn’t propagated yet, so nodes were repeatedly dropped. The fix was for the engineers to disable the system manually.6

There is a way to stop this from happening. It’s called a fail-open panic threshold: if more than N% of the fleet reports unhealthy, stop trusting the probe and stop evicting. It’s standard practice in load-balancing design. It was absent at Microsoft and effectively absent at AWS.

The most disheartening part of the Friday the 13th tech massacre is that we have seen all of this before, with reports on how to avoid them happening again.

A few of these stand out. The August 2003 Northeast blackout, which tragically killed around 100 people, was caused when FirstEnergy’s alarm system locked up, and the IT staff did not tell the operators. This caused the Zone 3 relays to trigger a cascading trip across most electrical lines.  Sound familiar? The losses totaled over $10 billion, and they produced a 240-page report with 46 recommendations.7

In July 2024, CrowdStrike introduced a malformed content update to all its endpoints, bringing down airports and other organizations nationwide. Our company was fortunate that our night techs saw the issue and deployed the fix within an hour. Other companies woke up to find that the system meant to protect their systems had brought them down. Again, sound familiar?

Why do we have to keep relearning these lessons? First, not as an excuse, these systems are extremely complicated. Second, the fixes to prevent these types of outages have to be re-implemented every time, and nothing forces people to build manual overrides or safeguards. To be honest, technology is moving too fast, and IT professionals are replacing entire systems every five to ten years. This constant cycle of new technology almost guarantees we will keep repeating these issues.

AI (Complex Information Processors) can help. They should be able to alert humans when events are unfolding more quickly and allow engineers to intervene sooner. However, the danger with AI is that it can, and probably will, cause issues itself. Those issues could be more damaging and even more widespread. We need to move slowly in our adaptation to AI, but unfortunately, history tells me that something terrible must happen before that becomes a reality.

Organizations have spent billions to prevent cyberattacks (which they should), only to watch their systems designed to prevent outages cause them. I hope we are learning, but as I type this, customers of several large banks are reporting that they cannot log into their accounts. [Sigh.]

 


1. FourWeekMBA, “Microsoft 365 Root Cause: Latent Bug Behind 36-Hour Outage,” 15 November 2026. https://fourweekmba.com/ai-microsoft-365-root-cause-latent-bug-behind-36-hour-outage/ (relays Microsoft’s final MO1550821 service-health entry; duration arithmetic is the article’s own) · See also Kate O’Flaherty, “Microsoft Apologises For Outage And Reveals Cause,” Forbes, 13 November 2026. https://www.forbes.com/sites/kateoflahertyuk/2026/11/13/microsoft-outage-what-happened-and-what-to-do-next/

2. KSAT, “Flights can resume after data overload on systems forced nationwide ground stop, FAA says,” 13 November 2026. https://www.ksat.com/news/local/2026/11/13/faa-issues-nationwide-groundstop/

3. Al Jazeera, “FAA ground stop lifted: Why were US flights grounded, and what’s next?”, 14 November 2026. https://www.aljazeera.com/news/2026/11/14/faa-ground-stop-lifted-why-were-us-flights-grounded-and-whats-next

4. CBS News, “FAA issues nationwide ground stop due to ‘extraordinary circumstances,’ White House says no indication of ‘nefarious act,’” 13 November 2026. https://www.cbsnews.com/news/faa-issues-nationwide-ground-stop/

5. Al Jazeera, “FAA ground stop lifted: Why were US flights grounded, and what’s next?”, 14 November 2026. https://www.aljazeera.com/news/2026/11/14/faa-ground-stop-lifted-why-were-us-flights-grounded-and-whats-next

6. Amazon Web Services, “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region,” October 2025. https://aws.amazon.com/message/101925 (vendor post-event summary — the primary document)

7. Practical Engineering, “What Really Happened During the 2003 Blackout,” 9 February 2022, summarizing the US–Canada Power System Outage Task Force final report. https://practical.engineering/blog/2022/2/9/what-really-happened-during-the-2003-blackout · Primary report: https://www.nerc.com/pa/rrm/ea/2003%20Blackout%20Final%20Report/ch6.pdf

 

Whether you’re looking for a dynamic partner on your next tech project, managed IT service providers, or are interested in joining our team of seriously awesome technicians — submit a contact form and we’ll be in touch!

Other blogs