Cases2 cases

1 to 2 of 2

1. IL-0378 All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storage

   Cloudflare2025

   Symptoms

   From 21:38 UTC on 21 March 2025, Cloudflare R2 object storage returned errors for 1 hour and 7 minutes: all object writes failed and about 35% of object reads failed globally. R2 global availability alerts fired at 21:45 UTC, at a 2% error budget burn rate.

   Credentials unavailableHigh error rate

   Trigger

   During a routine rotation of the credentials the R2 Gateway Worker uses to authenticate with the storage infrastructure, the new credentials were deployed to the Worker of the default environment instead of the production Worker, and the old credentials were then removed from the storage infrastructure at 20:37 UTC.

   Root cause

   Operator error

   Component

   Object storage

   Contributing factor

   Unverified assumptionMonitoring blind spotChange with a delayed effect

   Detection

   R2 global availability alerts fired at 21:45 UTC, seven minutes after impact began, at a 2% error budget burn rate, and the R2 engineering team turned to operational dashboards and logs.

   Remediation

   • Deployed the new credentials to the production R2 Gateway Worker, which restored R2 availability

   Action items

   • Add logging tags with the suffix of the credential ID the R2 Gateway Worker uses
   • Require confirmation that the new token ID suffix matches storage infrastructure logs before deleting the old token
   • Run key rotation through the hotfix release tooling, which enforces the environment, instead of manual wrangler commands
   • Write the two-person validation of these changes into the standard operating procedures
   • Extend the closed loop health check system to test new keys, report their status through alerting and confirm global propagation before releasing the gateway Worker
   • Add observability views of upstream success rates that bypass caching

   Lessons

   Because the deletion of the old credentials took effect gradually, availability metrics were a poor way to confirm the change. Cloudflare said the team should have checked directly which token the R2 Gateway service was using to authenticate with the storage infrastructure instead of relying on those metrics.

   View case
2. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'

   MIT Open Learning2023

   Symptoms

   On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

   Service outageCredentials unavailable

   Trigger

   A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

   Root cause

   Configuration change

   Component

   DatabaseSecrets store

   Contributing factor

   Unreviewed defaultsChange with a delayed effectConfiguration not in code

   Detection

   Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

   Remediation

   • Created a new database parameter group and applied it to the running RDS instance
   • Cleared the cached credentials and re-applied the Heroku configuration through Salt

   Action items

   • Move the management of xPro resources, including the RDS instance, into Pulumi, where AWS default parameter groups are not used

   Lessons

   Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

   View case

By root cause2