Incident Library

IL-0294 CircleCI · 2021-05-21databasegcpaws

CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

At 08:37 UTC on 20 May CircleCI upgraded its RabbitMQ cluster from 3.8.9 to 3.8.16 as routine maintenance, after checking the changelog for breaking changes. The changelog described a new 15 minute acknowledgement timeout for consumers that take messages without acknowledging them, but only for quorum queues, which CircleCI does not use. In fact the change, introduced in RabbitMQ 3.8.15, applied to all queue types. Messages that hit the timeout made RabbitMQ close the channel of the consumer, and on the queues that need a timeout that long, consumers dropped and did not come back. The queue that asynchronously cleans up VMs in one region lost consumers over time until it had none, and old VMs stopped being deleted.

CircleCI then hit its GCP CPU quotas, and automated retries kept the quotas exhausted, so new VMs could not be started; because the Docker executor fleet was coupled to the VM service, Docker jobs slowed too. Engineers blocked Linux machine jobs from the work queue at 14:13, GCP raised the quotas at 14:37, and engineers deleted old VMs by hand and cancelled some workflows at 17:22. At 16:25 they found the VM destroy queues had no consumers and reset the VM services deployment, and the queues drained. That load used up the Burst Balance of the RDS-based databases at about 18:05; a reboot did not help, so an RDS instance was scaled up at 18:20 and again at 19:15, while VM provisioning was turned down and remote-docker jobs were blocked to shed load.

The system was stuck in a cycle of creating VMs and being unable to remove them because of the GCP quotas. At 21:00 engineers scaled down all pods that created and destroyed VMs, at 21:57 deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues, and from 22:36 brought provisioning and distribution back one component at a time. By 00:39 everything was back online and the code changes made during the outage were reversed, and the incident was resolved at 01:18 UTC on 22 May. On 24 May graphs showing RabbitMQ consumers disconnected after 15 minutes revealed the leading cause, and RabbitMQ confirmed that the timeout affects all queue types and said its next release would apply a larger timeout and document the setting.

Severity
–
Scope
Docker, Machine executor (Windows, Mac, Arm, GPU) and remote Docker jobs delayed or blocked; macOS jobs slow
Users affected
CircleCI customers running these job types
Region
–
Time to detect
–
Time to resolve
11h 48m

Symptoms

From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

There had been signs earlier that day. At 02:31 engineers were told that a VM destroyer service queue had no consumers, and restarted the VM service Kubernetes pods. From 03:10 the queue lost consumers again without anyone noticing, and at 10:11 an alert reported that a queue used to destroy VMs was backing up, but it was set to notify only once at a threshold and did not page anyone.

A rise in cryptomining abuse of the platform looked like the cause, so engineers first removed the mining jobs, and the status page blamed platform abuse; this did not visibly reduce the impact. Meanwhile automated retries kept the GCP quotas exhausted, and when GCP raised the quotas at 14:37 the system still did not recover on its own.

After the VM destroy queues were reconnected at 16:25 and began to drain, the RDS-based databases used up their Burst Balance at about 18:05: IO dropped sharply and connections to the database timed out and dropped. Rebooting the database did not help, and later the GP2 credits ran out as well. Customers saw Machine Executor and Remote Docker jobs queued, in permanent not run status or canceled, and macOS jobs could be slow.

Symptom Throughput degradationCloud API rate limit exhaustedDatabase overload

Root cause

Root cause Breaking change in an upgrade

Component Message brokerDatabase
Contributing factor Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures
Trigger
The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

Detection

Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.