Anatomy of a Disaster: Why Cloudflare Went Dark — A Technical Breakdown of the November 18 Outage

Dead Air

On November 18, 2025, at exactly 11:20 UTC, network engineers around the world froze. Traffic graphs that normally pulse with activity suddenly nosedived, while error-code charts shot straight up. Users trying to reach their favorite sites, from e-commerce platforms to simple blogs, hit a wall: an "Internal Server Error" (Error 500). Discord was down, Shopify was struggling, and even Cloudflare's own status page seemed to be choking.

Many of us watched the event with growing unease. It adds to a grim string of outages this fall, which we covered in our roundup: Déjà Vu: Is the Internet Living Through Its "Autumn of the Middle Ages"?. We saw a similar paralysis a month earlier during the major Amazon Web Services outage.

Now that the dust has settled and the servers are humming along normally again (or at least spinning their fans at the usual pace), we can look under the hood of this incident. Thanks to the Cloudflare team's transparency, we know exactly what happened. And no, it wasn't a hacker attack, even though everything initially pointed that way. It was a lesson in humility for software engineering: the story of how a "safe" database permissions change and one unfortunate logic error in the code can bring the internet to a halt.

Join me on a technical journey inside Cloudflare's "black box." Grab a coffee. This one is long and detailed.

"It Has to Be an Attack!" – The Fog of War

Every large-scale incident looks like chaos in its early stages. When monitoring systems started lighting up red at 11:20, the first thought in Cloudflare's Network Operations Center (NOC) wasn't "a bug in the code." Suspicion fell on a massive DDoS attack.

Why? Because the symptoms were misleading. Systems didn't fail completely and all at once. Engineers saw strange fluctuations: traffic dropped, recovered, then dropped again. That's the classic picture of defensive systems fighting a botnet that keeps changing attack vectors. On top of that, engineers raised concerns in internal chats that this might be a continuation of activity by the "Aisuru" botnet, which had been flexing its muscles in recent weeks. Even Matthew Prince, Cloudflare's CEO, voiced the worry: "I worry this is the big botnet flexing."

Making matters worse, Cloudflare's status page (Cloudflare Status) went down too. In theory, it is hosted outside Cloudflare's infrastructure precisely so that it stays available during an outage. But the coincidence led engineers to suspect a sophisticated, targeted attack hitting multiple points at once.

Only a deeper look at the logs showed that the problem wasn't coming from outside. The source was at the very heart of the system.

Understanding ClickHouse: The Architecture of the Problem

To understand what really happened, we need to dive into the architecture of the databases Cloudflare uses. The company relies heavily on ClickHouse, a columnar OLAP database that is ideal for analyzing massive volumes of data in real time.

In ClickHouse, a cluster is made up of many shards. To query data across all shards at once, you use so-called Distributed tables. These tables usually live in a database named default. The physical data stored on individual nodes, however, sits in underlying tables, often in a database labeled r0 (or something similar, depending on convention).

Until the day of the outage, users (and systems) querying table metadata, such as a list of columns, saw only what was in the default database. Access to the r0 database was implicit.

The Change That Changed Everything

At 11:05 UTC, a change was deployed that was meant to improve security and reliability. Ironic, isn't it?

Engineers wanted distributed queries to run under the more granular permissions of the user who initiated the query. To do that, they changed the permissions configuration so that access to tables in r0 became explicit. This would let the system better enforce resource limits and permissions for individual subqueries. On paper, it was a great change, practically textbook, in the spirit of least privilege and tighter control.

In practice, it had one unintended side effect.

The Query That Killed the System

Cloudflare's platform includes a Bot Management module. It scores every HTTP request to determine whether it came from a human or a bot. The system relies on machine learning (ML) models, and those models need "features": specific traffic attributes they use to make their assessment.

The feature configuration isn't static. It is generated dynamically every 5 minutes based on traffic analysis, which lets Cloudflare adapt to new threats almost in real time.

The process that generates this configuration (the feature file) runs a query against ClickHouse to retrieve the list of available columns (features).

The critical flaw was that this query did not filter results by database name. Before 11:05, it returned the column list only for the table in the default database. After 11:05, once permissions on r0 became explicit, the same query started returning the column list twice: once for the table in the default database and again for the table in the r0 database.

Instead of a list of unique features, the system received a list full of duplicates.

Memory and a Fatal Error-Handling Choice

Now we move from the database layer to the application layer. Cloudflare's core proxy system (known internally as "FL," or "FL2" in its newer version) is written in Rust. Rust is famous for memory safety and performance, but it is also very strict about error handling.

To optimize performance, the Bot Management module in the proxy preallocates memory for features. Engineers set a hard limit on the number of features that could be handled. That limit was 200, well above actual usage at the time, which hovered around 60 features. It seemed like a safe margin: more than three times the headroom.

But once the database query started returning duplicate rows, the resulting configuration file ballooned. Because of the duplicates, the number of defined features exceeded the 200 limit.

This brings us to the heart of the problem in the code. The developer used a construct that assumed adding features to the buffer would always succeed. In programming, and especially in Rust, there is an instruction (commonly called unwrap) that tells the compiler: "Trust me, there won't be an error here, and if there is, stop the whole program."

Because the configuration file was malformed (too large), that instruction triggered a so-called panic. The thread handling network traffic simply "died."

The Death Loop and the Domino Effect

Had this happened once, the system would have restarted and recovered. The problem was that the corrupted configuration file was propagated to every server in Cloudflare's network.

1. The server pulls the new configuration (with duplicates).
2. The proxy tries to load it.
3. The code hits the 200-feature limit.
4. The process panics and terminates.
5. The supervisor restarts the proxy process.
6. The process comes back up, pulls the configuration... and we're back to step 1.

The result was a global restart loop across thousands of machines. To end users, it looked like a 500, 502, or 504 error, because the proxy couldn't serve the request.

Interestingly, the older version of the proxy ("FL") didn't panic. Instead, the bad configuration simply caused it to malfunction, assigning all traffic a bot score of zero. That meant customers relying on bot blocking could be flooded with spam, or, if they had rules blocking low scores, end up blocking real users.

Why Did the Dashboard and Other Services Go Down?

Cloudflare is a system of interconnected parts. The failure of the core proxy dragged other services down with it:

• Workers KV: This service is the foundation of many Cloudflare systems. Because it is accessed through the core proxy, the avalanche of errors cut off access to its data.
• Cloudflare Access: This service depends on Workers KV. No KV meant no login or authorization.
• Dashboard (customer portal): Cloudflare's admin dashboard uses Turnstile (its CAPTCHA alternative) at login. Because Turnstile wouldn't load (the proxy was down), no one could log in to see what was going on. Even Cloudflare's own staff had trouble with internal tools.

It's a textbook example of a circular dependency in a crisis.

The Road to Recovery: From Chaos to Stability

The repair process was far from trivial. Remember the fog of war.

1. 11:20 - 13:05: The team fought the symptoms, believing it was a KV performance problem or an attack. They tried rerouting traffic and rate-limiting accounts.
2. 13:05: Breakthrough. The team deployed a bypass for Workers KV and Access so that these services skipped the broken core proxy. This let engineers regain control of their tools and reduced the impact on key internal systems.
3. 13:37: The culprit was identified: the Bot Management configuration file. The team understood that it was causing the failure in the code.
4. 14:24: Automatic generation and propagation of new (faulty) files was stopped.
5. 14:30: The last known good version of the configuration file was manually injected into the distribution system.
6. Restart: The proxy was force-restarted. Because the file was now valid (within the 200 limit), the processes came up cleanly and stopped restarting.

Until 17:06, engineers were cleaning up after the outage, restarting the remaining services that had fallen into an error state during the main incident.

Takeaways: A Lesson for the Entire Industry

The Cloudflare incident is a harsh reminder of several fundamental principles of distributed systems engineering that are easy to forget in the rush to innovate.

First: Input validation isn't just about protecting yourself from users. Cloudflare treated data from its own database as "trusted." The code assumed the database would return unique columns because "it always had." A change in another part of the system broke that assumption. Treat data from internal systems with the same suspicion as user input.

Second: Errors shouldn't kill the process. In production code, a panic in the main thread handling network requests is unacceptable. A configuration error should result in logging the error and possibly running in a degraded mode, never in a complete service shutdown.

Third: Observability under extreme conditions. The fact that debugging systems consumed huge amounts of resources during the outage (while trying to report errors) made things worse by increasing latency. Monitoring systems must not overload the very system they are trying to diagnose.

Fourth: Gradual rollouts. Both the database change that caused the problem and the propagation of the faulty file had global effects. The lack of a staged rollout mechanism (canary release) for configuration files meant the error hit everyone at once. Cloudflare has already announced that it will test configuration files with the same procedures it uses for code.

Summary

Fall 2025 will go down as a dark chapter in internet history. After the AWS outage in October, Cloudflare has now joined the infamous "Total Paralysis" club. It shows how heavily our digital world depends on just a handful of pillars. When one of them wobbles, whether because of a malicious attack or a single redundant database row, the entire global economy shakes.

What happened on November 18 wasn't the work of hackers. It was human error, process failure, and a bug in the code. As mundane as that sounds, at the scale Cloudflare operates, the mundane turns into catastrophe.

Will it happen again? Certainly. Not necessarily at Cloudflare, and not necessarily in the same way. But systems this complex are inherently prone to failure. Our job, as security professionals and administrators, is to build resilient systems that can survive the fall of a giant. Or at least to have a plan B ready for the day we suddenly see Error 500 on half the sites we visit.

For more context on this outage, read our article: Déjà Vu: Is the Internet Living Through Its "Autumn of the Middle Ages"?. We also recommend our analysis of cloud risk in strategic sectors.

Stay safe, and... check your error handling!

Source: Official Cloudflare blog.

Aleksander