Incident Library

IL-0283 Cloudflare · 2025-11-18databasenetworking

Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

Every request to Cloudflare passes the HTTP and TLS layer, then the core proxy FL, then Pingora for cache lookups or origin fetches. The core proxy applies each customer's configuration through modules, one of which is Bot Management: a machine learning model gives every request a bot score, using a feature configuration file that is refreshed every few minutes and published to the entire network. A query on a ClickHouse cluster generates this file every five minutes. In that cluster, distributed tables in the database default read the underlying tables of each shard in the database r0, and queries to the distributed tables ran through a shared system account; work was under way to run them under the initial user accounts so limits and grants could be applied per user.

At 11:05 UTC a permissions change made users' access to the r0 tables explicit. The feature file query read name and type from system.columns for the table 'http_requests_features' without filtering on the database name, on the old assumption that only the default database would be returned. After the change it also returned the r0 columns, more than doubling the rows and so the features in the file. The proxy preallocates memory for features and limits them to 200, against about 60 in use, so when the file with more than 200 features reached the servers, the FL2 Rust code hit an unhandled error and panicked, and the proxy returned 5xx. Since only the updated ClickHouse nodes produced bad files, the network failed and recovered every few minutes until all nodes produced the bad file.

At 13:05 Cloudflare used internal bypasses so Workers KV and Access fell back to a prior proxy version, which reduced their impact and the errors of services that rely on Workers KV. By 13:37 the team was confident the Bot Management file was the trigger and worked on restoring a last-known-good version. At 14:24 it stopped the creation and propagation of new feature files and confirmed recovery with the old file; at 14:30 a correct file was deployed globally, by inserting a known good file into the distribution queue and forcing a restart of the core proxy, and core traffic largely flowed normally. A second period of reduced dashboard availability, from 14:40 to 15:30, came from a backlog of login attempts and retries and ended when control plane concurrency was scaled. Remaining services that had entered a bad state were restarted, and 5xx volume returned to normal at 17:06. Cloudflare called it its worst outage since 2019.

Severity
–
Scope
Core CDN and security services returned HTTP 5xx errors and the majority of core traffic stopped flowing; Turnstile, Workers KV, Access, dashboard login and parts of Email Security were also affected, and on the old FL proxy all traffic got a bot score of zero
Users affected
Internet users visiting Cloudflare customers' sites and Cloudflare customers
Region
Cloudflare's whole network
Time to detect
3m
Time to resolve
5h 38m

Symptoms

From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

The errors came from the core proxy, called FL or Frontline, for all traffic that depended on its Bot Management module. On the new proxy engine, FL2, the module panicked with "thread fl2_worker_thread panicked: called Result::unwrap() on an Err value" and requests failed with 5xx. Customers still on the old FL engine saw no errors, but every request got a bot score of zero, so rules that block bots produced large numbers of false positives. CDN response latency also rose sharply, because debugging and observability systems spent large amounts of CPU adding debugging information to uncaught errors. Workers KV returned significantly elevated 5xx errors, Access authentication failed for most users, Turnstile failed to load, and most users could not log in to the dashboard.

At first the failures came and went: the network recovered for a while and then failed again. The feature file was regenerated every five minutes by a query on a ClickHouse cluster that was being updated gradually, and only runs on the updated part produced a bad file, so good and bad files alternated across the network. In the end every ClickHouse node produced the bad file and the network stayed in the failing state.

Because this fluctuation was very unusual for an internal error, the team first suspected a hyper-scale DDoS attack, possibly a continuation of the recent high-volume Aisuru DDoS attacks. Cloudflare's status page, which is hosted entirely off Cloudflare's infrastructure, went down at the same time by coincidence, and some responders took it as a sign that an attacker was targeting both. The first symptom the team investigated was a degraded Workers KV response rate, and from 11:32 to 13:05 it tried traffic manipulation and account limiting to bring Workers KV back to normal.

Symptom High error rateService outageHigh latencyResponse quality degradation

Root cause

Root cause Configuration change

Component DatabaseCDN and edge network
Contributing factor Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm
Trigger
At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

Detection

The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.