Incident Library

IL-0328 CircleCI · 2021-11-08database

A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"

All work coming into CircleCI's execution platform goes through a distribution service that accepts requests to run a task and moves each task to the right queue, using a PostgreSQL database with schema validation. When the distributor finds bad data while looking for work, it stops and retries from the start, on the assumption that a retry will be enough. A deploy at 18:50 UTC changed a field type in that database; new work arrived with the new type, and work queued before the deploy made distribution fail. The team rolled the change back, but then the data written between the two deploys became unreadable and distribution kept failing. The team wrote, built and manually deployed a change that made the distribution system ignore the field.

To absorb the expected rush of queued work, the "thundering herd", the team manually scaled the primary compute cluster, a standard step in its incident process. New nodes took longer than expected to join, many were marked unhealthy and terminated, and machine executors including Mac were slowed by provider rate limiting, so recovery took longer than it should have. Queues drained over time, and after a period of monitoring with normal processing the incident was resolved. CircleCI found gaps in its testing strategy and said it would change how the distributor reacts to unexpected data so that such a deploy should not affect every job.

Severity
–
Scope
Customer jobs on all executors were blocked in a "not running" state from about 18:50 to 19:43 UTC, and queue times stayed longer than usual, particularly for machine executors, until 20:19 UTC
Users affected
CircleCI customers
Region
–
Time to detect
–
Time to resolve
1h 29m

Symptoms

From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

The error count chart showed errors spiking first, then incoming work falling as jobs that depend on other jobs could not be queued. Rolling the change back did not clear the errors: they changed form but stayed, because the data written between the two deploys had become unreadable, and incoming work kept falling slowly. The distributor is built to stop when it finds bad data and to retry from the start.

Docker executors recovered first. After the fix, queue times stayed higher than usual for longer than expected, especially for machine executors, until 20:19 UTC. New nodes added to the primary compute fleet took longer than expected to join, and many were marked unhealthy and terminated, and machine executors, including Mac, were provisioned more slowly because of provider rate limiting.

Symptom Service outageHigh latencyScale-up failureCloud API rate limit exhausted

Root cause

Root cause Faulty deployment

Component DatabaseNode pool
Contributing factor Ineffective rollbackExcessive blast radiusGap in automated tests
Trigger
At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

Detection

Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Node pool among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.