Cases3 cases

1 to 3 of 3

1. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day

   Mercari2026

   Symptoms

   From about 6:00 on the last business day of a month, Cloud Pub/Sub messages of the Merpay accounting event pipeline began to back up, new accounting data stopped being registered in Cloud Spanner, and the accounting system intermittently could not accept new data. The backlog grew to about 10 million messages.

   Throughput degradationDatabase overloadAutoscaler misbehaviour

   Trigger

   More than a month earlier the Cloud Functions of the pipeline had been moved from 1st gen to 2nd gen (based on Cloud Run) without setting --max-instances in the deploy command, which lowered the scale limit from effectively unlimited, the 1st gen default, to 100, the 2nd gen default. The month-end peak, which on that day continued for about 40 minutes, reached that limit.

   Root cause

   Breaking change in an upgrade

   Component

   Serverless function platformMessage brokerDatabase

   Contributing factor

   Unreviewed defaultsUnknown limitChange with a delayed effectMonitoring blind spot

   Remediation

   • Re-ingested the messages saved to GCS by the Dataflow subscription, avoiding data loss

   Action items

   • Set max-instances and the other scaling parameters of Cloud Run explicitly in the deploy definition, raising max-instances from 100 to 1000 in steps
   • Set min-instances to 1 to avoid cold starts, raise concurrency from 1 to 10, and set CPU to 1 and memory from 512MB to 1Gi
   • Use the dual CPU scaling mode of spanner-autoscaler v0.8.0 with a new Total CPU threshold of 70%
   • Lower the High priority CPU threshold from 60% to 55% so scale-up starts earlier
   • Forbid Spanner scale-down between 5:00 and 7:00
   • Reserve Spanner processing units on a schedule for periods with predictable load
   • Alert in Datadog when the Cloud Run instance count reaches 40% and 60% of its limit
   • Alert in Datadog when undelivered Pub/Sub messages exceed a threshold, lowering it step by step

   Lessons

   When moving to a new generation of a managed service, defaults that the team never changed can surface as problems under load, so parameters that only matter at peak, such as scale limits, concurrency and timeouts, should be set explicitly at every migration. An outage rarely has one cause; the fact that the previous month passed under the same conditions pointed to traffic continuity, and prevention needs to look at the trigger, the amplification and what turned it into damage. Capacity settings and the monitoring that shows when they are approached have to be designed together, and teams that run systems they inherited should review these settings and assumptions regularly.

   View case
2. IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growing

   Kanmu2024

   Symptoms

   A Datadog alert fired because a large number of unprocessed jobs had piled up in Kanmu's job queue and were not being worked off. The jobs were the batch enqueued at about the same time every day, in the usual volume, but the workers took far longer than normal to dequeue and process them.

   Throughput degradationHigh latency

   Trigger

   A heavy ad hoc analysis query from the BI tool Redash ran for a long time on the writer instance, because the Redash data source pointed at the cluster endpoint.

   Root cause

   Operator error

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Change with a delayed effectUnreviewed defaults

   Detection

   A Datadog alert on unprocessed jobs piling up in the job queue.

   Remediation

   • Changed the Redash data source to the reader endpoint
   • Ended the long-running Redash query with pg_terminate_backend
   • Ran VACUUM on que_jobs by hand again, which cleaned up the dead tuples

   Action items

   • Make Redash unable to connect to the writer instance even if the writer endpoint is set, by mapping its host name to a TEST-NET-1 address in the Redash container's /etc/hosts
   • Set statement_timeout on PostgreSQL users so that long queries stop automatically
   • Consider improving qg's dequeue algorithm or adopting a job queue such as river that uses SELECT FOR UPDATE SKIP LOCKED

   Lessons

   A long-running query on the same database instance, even one unrelated to que_jobs, can keep VACUUM from removing que_jobs dead tuples, so the table keeps growing and the job queue slows down. qg's design with advisory locks and a recursive CTE is sensitive to dead tuples.

   View case
3. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

   CircleCI2021

   Symptoms

   From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

   Throughput degradationCloud API rate limit exhaustedDatabase overload

   Trigger

   The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

   Root cause

   Breaking change in an upgrade

   Component

   Message brokerDatabase

   Contributing factor

   Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures

   Detection

   Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

   Remediation

   • Blocked Linux machine jobs from entering the work queue
   • Had GCP raise the CPU quotas
   • Deleted old VMs by hand
   • Cancelled some workflows to free resources
   • Reset the VM services deployment so the VM destroy queues had consumers again
   • Scaled up the RDS database instance, at 18:20 and again at 19:15
   • Turned down VM provisioning and blocked remote-docker jobs to shed load
   • Scaled down all pods that created and destroyed VMs
   • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
   • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes

   Action items

   • Make VM provisioning more robust to changes in load so it can recover from thundering herds
   • Make VM provisioning independent so its failure does not affect Docker jobs
   • Review and improve observability of RabbitMQ and the production queues
   • Batch requests to avoid hitting GCP API rate limits
   • Improve incident response for long-running incidents

   View case

By root cause2