Cases2 cases

1 to 2 of 2

1. IL-0378 All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storage

   Cloudflare2025

   Symptoms

   From 21:38 UTC on 21 March 2025, Cloudflare R2 object storage returned errors for 1 hour and 7 minutes: all object writes failed and about 35% of object reads failed globally. R2 global availability alerts fired at 21:45 UTC, at a 2% error budget burn rate.

   Credentials unavailableHigh error rate

   Trigger

   During a routine rotation of the credentials the R2 Gateway Worker uses to authenticate with the storage infrastructure, the new credentials were deployed to the Worker of the default environment instead of the production Worker, and the old credentials were then removed from the storage infrastructure at 20:37 UTC.

   Root cause

   Operator error

   Component

   Object storage

   Contributing factor

   Unverified assumptionMonitoring blind spotChange with a delayed effect

   Detection

   R2 global availability alerts fired at 21:45 UTC, seven minutes after impact began, at a 2% error budget burn rate, and the R2 engineering team turned to operational dashboards and logs.

   Remediation

   • Deployed the new credentials to the production R2 Gateway Worker, which restored R2 availability

   Action items

   • Add logging tags with the suffix of the credential ID the R2 Gateway Worker uses
   • Require confirmation that the new token ID suffix matches storage infrastructure logs before deleting the old token
   • Run key rotation through the hotfix release tooling, which enforces the environment, instead of manual wrangler commands
   • Write the two-person validation of these changes into the standard operating procedures
   • Extend the closed loop health check system to test new keys, report their status through alerting and confirm global propagation before releasing the gateway Worker
   • Add observability views of upstream success rates that bypass caching

   Lessons

   Because the deletion of the old credentials took effect gradually, availability metrics were a poor way to confirm the change. Cloudflare said the team should have checked directly which token the R2 Gateway service was using to authenticate with the storage infrastructure instead of relying on those metrics.

   View case
2. IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growing

   Kanmu2024

   Symptoms

   A Datadog alert fired because a large number of unprocessed jobs had piled up in Kanmu's job queue and were not being worked off. The jobs were the batch enqueued at about the same time every day, in the usual volume, but the workers took far longer than normal to dequeue and process them.

   Throughput degradationHigh latency

   Trigger

   A heavy ad hoc analysis query from the BI tool Redash ran for a long time on the writer instance, because the Redash data source pointed at the cluster endpoint.

   Root cause

   Operator error

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Change with a delayed effectUnreviewed defaults

   Detection

   A Datadog alert on unprocessed jobs piling up in the job queue.

   Remediation

   • Changed the Redash data source to the reader endpoint
   • Ended the long-running Redash query with pg_terminate_backend
   • Ran VACUUM on que_jobs by hand again, which cleaned up the dead tuples

   Action items

   • Make Redash unable to connect to the writer instance even if the writer endpoint is set, by mapping its host name to a TEST-NET-1 address in the Redash container's /etc/hosts
   • Set statement_timeout on PostgreSQL users so that long queries stop automatically
   • Consider improving qg's dequeue algorithm or adopting a job queue such as river that uses SELECT FOR UPDATE SKIP LOCKED

   Lessons

   A long-running query on the same database instance, even one unrelated to que_jobs, can keep VACUUM from removing que_jobs dead tuples, so the table keeps growing and the job queue slows down. qg's design with advisory locks and a recursive CTE is sensitive to dead tuples.

   View case

By root cause1