Incident Library

IL-0378 Cloudflare · 2025-03-21database

All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storage

R2 is made of three parts: the production gateway Worker that serves the S3 API, REST API and Workers API, a metadata service, and storage infrastructure that holds the encrypted object data. The R2 Gateway Worker authenticates to the storage infrastructure with a credential ID and key pair that Cloudflare rotates regularly. Production and development are separate Workers environments, each an isolated Worker with its own environment variables and secrets, and both the wrangler secret put and wrangler deploy commands use the default environment when the --env parameter is left out.

On 21 March 2025 the R2 engineering team began a rotation at 19:49 UTC by creating new credentials while keeping the old ones. At 20:19 UTC the team set the new secret and deployed the gateway, but both commands were run without --env production, so the new credentials reached the Worker of the default environment while the production Worker kept the old ones. Believing production had been updated, the team removed the old credentials from the storage infrastructure at 20:37 UTC. As that deletion propagated, the production R2 Gateway Worker could no longer authenticate with the storage backend, and from 21:38 UTC writes failed and many reads failed; an intermediate cache in front of storage kept serving a large share of reads.

After alerts at 21:45 UTC and an internal incident declared at 21:50 UTC, the team suspected a credential propagation regression and twice made new credentials, the second set deployed again to the wrong Worker. At 22:36 UTC a review of the production Worker's release history showed that the credentials had gone to a non-production Worker. The team deployed them to the production Worker and R2 availability recovered at 22:45 UTC. No data was lost or corrupted: uploads and changes that had returned a successful HTTP status were stored. Cloudflare said the incident came from human error and lasted longer because it could not see which credentials the gateway Worker was using.

Severity
–
Scope
R2 object writes (100% failed) and object reads (about 35% failed globally), and services that use R2: Billing invoices, Cache Reserve, Email Security metrics, Images, Key Transparency Auditor, Log Delivery, Stream and Vectorize
Users affected
All customers using Cloudflare R2
Region
Global
Time to detect
7m
Time to resolve
1h 7m

Symptoms

From 21:38 UTC on 21 March 2025, Cloudflare R2 object storage returned errors for 1 hour and 7 minutes: all object writes failed and about 35% of object reads failed globally. R2 global availability alerts fired at 21:45 UTC, at a 2% error budget burn rate.

Operations that touch only metadata, such as head and list, kept working, and cached object reads were not affected, so customers reading public assets through custom domains saw fewer errors. Services built on R2 failed with it: all Images and Stream uploads failed, successful delivery of stored images fell to about 25%, Stream video segment delivery fell to 94% and Stream Live was down, all Key Transparency Auditor operations failed, all Vectorize insert and upsert operations failed, Logpush and Logpull log delivery was delayed by up to 70 minutes, Cache Reserve sent more requests to customer origins, customers could get errors when downloading past invoices, and Email Security metrics stopped updating.

The drop in availability was gradual and not obvious at first, because the deletion of the old credentials reached the storage infrastructure with a delay. Seeing a slow but steady decline in both read and write availability, the R2 engineering team suspected a regression in how new credentials propagate through the storage infrastructure. At 22:15 UTC the team created another set of credentials to force propagation again, with no improvement, and at 22:30 UTC it deployed a fresh set to the R2 Gateway Worker to check whether the credentials themselves were at fault; that deploy also lacked the environment parameter and went to the same non-production Worker.

Symptom Credentials unavailableHigh error rate

Root cause

Root cause Operator error

Component Object storage
Contributing factor Unverified assumptionMonitoring blind spotChange with a delayed effect
Trigger
During a routine rotation of the credentials the R2 Gateway Worker uses to authenticate with the storage infrastructure, the new credentials were deployed to the Worker of the default environment instead of the production Worker, and the old credentials were then removed from the storage infrastructure at 20:37 UTC.

Detection

R2 global availability alerts fired at 21:45 UTC, seven minutes after impact began, at a 2% error budget burn rate, and the R2 engineering team turned to operational dashboards and logs.

Remediation

Action items

Lessons

Because the deletion of the old credentials took effect gradually, availability metrics were a poor way to confirm the change. Cloudflare said the team should have checked directly which token the R2 Gateway service was using to authenticate with the storage infrastructure instead of relying on those metrics.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.