Incident Library
Applied filters SymptomNode disk fullFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes
    CircleCI2026
    Symptoms

    On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.

    High latencyService outageThroughput degradationNode disk full
    Trigger

    Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.

    Root cause
    Capacity exhaustion
    Component
    DatabasePostgreSQL vacuumCI system
    Contributing factor
    Ignored early signalNo alert on limitSingle point of failure
    Detection

    Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.

    Remediation
    • Changed the anti-wraparound autovacuum settings to help it finish faster
    • Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
    • Changed database parameters to stop further maintenance runs from starting
    • Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
    • Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster
    Action items
    • Alert on how full the transaction log volume is and on how far archiving has fallen behind
    • Alert on the database maintenance that drives transaction log growth
    • Review all alerts so that noisy or sensitive alerts do not hide real issues
    • Tune how often and how aggressively autovacuum runs on the database
    • Track the size of the largest tables over time so they stay within safe limits
    • Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
    • Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
    • Include specific impact and timing sooner in status page updates during long incidents
    Lessons

    CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.

    View case
Check your cluster for this

This case involves PostgreSQL vacuum and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
PostgreSQL vacuumCI systemDatabase
Contributing factor
Ignored early signalNo alert on limitSingle point of failure
Organisation
CircleCI
  1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes
    CircleCI2026