Incident Library

IL-0284 Cloudflare · 2025-06-12database

Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

Workers KV is built as a "coreless" service that runs independently in each Cloudflare location, but it relies on a central data store as the source of truth, and part of that storage infrastructure is backed by a third-party cloud provider. Workers KV was being moved to more resilient infrastructure for its central store, including a migration to Cloudflare R2, and had removed a storage provider during that re-architecture; Cloudflare says there was a gap in coverage, which this incident exposed. Cloudflare builds its products on its own platform, so many of them rely on Workers KV for configuration, authentication and asset delivery, and SQLite-backed Durable Objects and D1 share the same underlying storage. When the third-party provider had an outage, cold reads and writes to the KV namespaces used across Cloudflare failed completely, and the failure cascaded into every product built on Workers KV. Magic Transit, Magic WAN, DNS, Cache, the proxy and WAF were not directly affected, and no data was lost.

The incident started at 17:52 UTC with failing WARP device registrations. At 18:06 the separate service incidents were merged once the shared cause was found, and at 18:21 the priority was raised from P1 to P0. From 18:43 Access worked with the Workers KV team on moving to a different backing datastore in case the storage stayed down, and from 19:45 teams worked on a Workers KV release against an alternative datastore. At 19:09 Gateway began degrading rules that referenced identity or device posture, and at 19:32 Access and Device Posture force-dropped identity and device posture requests to shed load on Workers KV. Turnstile and Challenges kill switches were turned on so visitors were not blocked; while they were on, siteverify could redeem valid tokens more than once. At 20:23 services began to recover with the storage infrastructure, at 20:25 Access and Device Posture resumed calling Workers KV, and at 20:28 Service Level Objectives were back at pre-incident levels. Cloudflare gave the outage's length as 2 hours and 28 minutes.

Severity
–
Scope
Workers KV failed 90.22% of requests; Access, Gateway, WARP, Dashboard, Challenges and Turnstile, Browser Isolation, Images, Stream, Realtime, Workers AI, Pages and Workers Assets, AutoRAG, Durable Objects, D1, Queues and Event Notifications, AI Gateway, parts of the CDN, Workers, Workers Builds, Browser Rendering and Zaraz were impaired; no data was lost
Users affected
all Cloudflare customers using the affected services
Region
global
Time to detect
–
Time to resolve
2h 36m

Symptoms

At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

Workers KV failed 90.22% of requests. Any key that was not in the cache and had to be read from Workers KV's origin storage backends failed with response code 503 or 500, while requests served from the cache still returned 200 or 404. Access, which keeps application and policy configuration and user identity in Workers KV and fails closed, failed 100% of identity-based logins. Gateway could not evaluate identity-based DNS over HTTPS rules, and users of the Gateway proxy, egress and TLS decryption could not connect. No new WARP clients could connect or sign up, and the WARP emergency disconnect override was itself unavailable because it also depends on Workers KV. Dashboard logins failed, the siteverify API of Challenges and Turnstile failed or timed out, all Workers AI inference requests failed, Stream errors exceeded 90%, Pages errors peaked at about 100%, Queues were unavailable, AI Gateway errors peaked at 97%, and SQLite-backed Durable Objects and D1, which share the same storage infrastructure, peaked at 22% errors.

The outage also spread to the CDN. Registration requests from Zero Trust clients increased sharply and loaded several Cloudflare locations, and automated traffic management moved CDN traffic to nearby locations. Part of the traffic was not rerouted as expected and saw elevated latency, HTTP 499 errors and HTTP 503 errors in São Paulo, Philadelphia, Atlanta and Raleigh. When the storage infrastructure started to recover at 20:23, a non-negligible error rate and infrastructure rate limits remained while services repopulated their caches.

Symptom High error rateService outageCredentials unavailable

Root cause

Root cause Cloud provider outage

Component Database
Contributing factor Single point of failureExcessive blast radiusCircular dependencyRetry storm
Trigger
Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

Detection

The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

Remediation

Action items

Lessons

Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.