Cases2 cases

1 to 2 of 2

1. IL-0284 Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

   Cloudflare2025

   Symptoms

   At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

   High error rateService outageCredentials unavailable

   Trigger

   Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Single point of failureExcessive blast radiusCircular dependencyRetry storm

   Detection

   The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

   Remediation

   • Activated the Challenges and Turnstile kill switches so visitors were not blocked
   • Degraded Gateway rules that referenced identity or device posture state to remove the Workers KV dependency
   • Force-dropped Access identity and device posture requests to shed load on Workers KV
   • Worked on a Workers KV release against an alternative backing datastore in case the storage stayed down
   • Restored Access and Device Posture calls to Workers KV once the third-party service was back

   Action items

   • Avoid singular dependencies on storage infrastructure that Cloudflare does not own
   • Improve the ability to recover critical services, including Access, Gateway and WARP
   • Add redundancy to Workers KV's storage infrastructure and remove the dependency on any single provider
   • Make each affected product resilient to the loss of any single point of failure, third-party dependencies included
   • Build tooling to re-enable namespaces progressively during storage incidents, so services come back without overloading Cloudflare's own infrastructure

   Lessons

   Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

   View case
2. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'

   MIT Open Learning2023

   Symptoms

   On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

   Service outageCredentials unavailable

   Trigger

   A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

   Root cause

   Configuration change

   Component

   DatabaseSecrets store

   Contributing factor

   Unreviewed defaultsChange with a delayed effectConfiguration not in code

   Detection

   Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

   Remediation

   • Created a new database parameter group and applied it to the running RDS instance
   • Cleared the cached credentials and re-applied the Heroku configuration through Salt

   Action items

   • Move the management of xPro resources, including the RDS instance, into Pulumi, where AWS default parameter groups are not used

   Lessons

   Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

   View case

By root cause2