Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0547 Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index
    Buildkite2025
    Symptoms

    From 04:48 UTC on 10 November 2025, queries on the table that stores annotations in Buildkite Pipelines timed out, CPU load on the affected databases rose sharply, and the Agent API, the REST API, GraphQL and the Web UI slowed down and returned more errors.

    Intermittent timeoutsDatabase overloadHigh error rateHigh latency
    Trigger

    At 04:48 UTC a database migration removed an index on the annotations table of the Pipelines product.

    Root cause
    Faulty deployment
    Component
    Database
    Contributing factor
    Missed in reviewNo staged rollout
    Detection

    At 04:53 UTC the impact of the removed index was detected as queries timing out on the annotations table.

    Remediation
    • Recreated the removed index, finishing on about half of the database shards by 05:30 UTC
    • Temporarily revoked the application user's permission to read from and write to the annotations table
    • Increased max_parallel_maintenance_workers in Postgres to speed up index creation
    • Finished recreating the index on the remaining shards between 07:50 and 08:43 UTC
    • Freed the agents and cleaned up the job states left inconsistent, handling by hand the cases the system could not self-heal, by 11:56 UTC
    Action items
    • Extend the internal tooling that flags risky migrations in code review to cover index removals and ask engineers to verify an index is no longer needed
    • Roll out high-risk migrations to shards in phases instead of to all shards at once
    • Evaluate tools such as HypoPG to analyse index creation and removal
    • Show index usage data from pganalyze more prominently in the development workflow
    • Automate the manual cleanup of inconsistent agent states so the system can heal itself
    Lessons

    Buildkite said the incident was mainly caused by a database migration that exposed gaps in its review process.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Database
Contributing factor
No staged rolloutMissed in review
Organisation
Buildkite
  1. IL-0547 Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index
    Buildkite2025