Cases5 cases

1 to 5 of 5

1. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case
2. IL-0284 Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

   Cloudflare2025

   Symptoms

   At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

   High error rateService outageCredentials unavailable

   Trigger

   Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Single point of failureExcessive blast radiusCircular dependencyRetry storm

   Detection

   The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

   Remediation

   • Activated the Challenges and Turnstile kill switches so visitors were not blocked
   • Degraded Gateway rules that referenced identity or device posture state to remove the Workers KV dependency
   • Force-dropped Access identity and device posture requests to shed load on Workers KV
   • Worked on a Workers KV release against an alternative backing datastore in case the storage stayed down
   • Restored Access and Device Posture calls to Workers KV once the third-party service was back

   Action items

   • Avoid singular dependencies on storage infrastructure that Cloudflare does not own
   • Improve the ability to recover critical services, including Access, Gateway and WARP
   • Add redundancy to Workers KV's storage infrastructure and remove the dependency on any single provider
   • Make each affected product resilient to the loss of any single point of failure, third-party dependencies included
   • Build tooling to re-enable namespaces progressively during storage incidents, so services come back without overloading Cloudflare's own infrastructure

   Lessons

   Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

   View case
3. IL-0378 All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storage

   Cloudflare2025

   Symptoms

   From 21:38 UTC on 21 March 2025, Cloudflare R2 object storage returned errors for 1 hour and 7 minutes: all object writes failed and about 35% of object reads failed globally. R2 global availability alerts fired at 21:45 UTC, at a 2% error budget burn rate.

   Credentials unavailableHigh error rate

   Trigger

   During a routine rotation of the credentials the R2 Gateway Worker uses to authenticate with the storage infrastructure, the new credentials were deployed to the Worker of the default environment instead of the production Worker, and the old credentials were then removed from the storage infrastructure at 20:37 UTC.

   Root cause

   Operator error

   Component

   Object storage

   Contributing factor

   Unverified assumptionMonitoring blind spotChange with a delayed effect

   Detection

   R2 global availability alerts fired at 21:45 UTC, seven minutes after impact began, at a 2% error budget burn rate, and the R2 engineering team turned to operational dashboards and logs.

   Remediation

   • Deployed the new credentials to the production R2 Gateway Worker, which restored R2 availability

   Action items

   • Add logging tags with the suffix of the credential ID the R2 Gateway Worker uses
   • Require confirmation that the new token ID suffix matches storage infrastructure logs before deleting the old token
   • Run key rotation through the hotfix release tooling, which enforces the environment, instead of manual wrangler commands
   • Write the two-person validation of these changes into the standard operating procedures
   • Extend the closed loop health check system to test new keys, report their status through alerting and confirm global propagation before releasing the gateway Worker
   • Add observability views of upstream success rates that bypass caching

   Lessons

   Because the deletion of the old credentials took effect gradually, availability metrics were a poor way to confirm the change. Cloudflare said the team should have checked directly which token the R2 Gateway service was using to authenticate with the storage infrastructure instead of relying on those metrics.

   View case
4. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours

   Cloudflare2023

   Symptoms

   From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

   DNS resolution failuresHigh error rate

   Trigger

   On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

   Root cause

   Software defect

   Component

   Recursive DNS resolver service

   Contributing factor

   Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests

   Detection

   External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

   Remediation

   • Deployed an override rule meant to stop 1.1.1.1 from answering with the stale root zone, which did not cover forwarded queries
   • Stopped 1.1.1.1 from preloading the root zone file entirely

   Action items

   • Add alerts for when static_zone serves a stale root zone file
   • Re-evaluate how the root zone is ingested and distributed internally so new record types are handled
   • Improve test coverage for what happens when the root zone fails to parse
   • Limit how long stale copies of the root zone can be used, following RFC 8806

   Lessons

   Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

   View case
5. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case

By root cause5