Incident Library

IL-0547 Buildkite · 2025-11-10database

Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index

At 04:48 UTC on 10 November 2025 Buildkite applied a database migration that removed an index on the annotations table of its Pipelines product. Another index was already in place to replace it, but a small number of high-traffic queries did not use the new index as expected. From 04:53 UTC those queries timed out, the agent command that creates annotations was behind many of them, and the CPU load they put on the affected PostgreSQL databases degraded every Pipelines feature. Migrations at Buildkite apply to all shards at the same time.

The team began recreating the removed index at once and finished it on about half of the shards by 05:30 UTC, which partly restored service. On the remaining shards the query load prevented efficient index creation. The team split up to try several ways of reducing that load: disabling annotations in the product, sending new annotations to a separate table, and blocking the table through table-level permissions. Temporarily revoking the application user's read and write permissions on the annotations table, together with raising max_parallel_maintenance_workers in Postgres, made index creation much faster, and between 07:50 and 08:43 UTC the index was rebuilt on the remaining shards, after which latency and error rates were back to normal.

Some agents and jobs were left in an inconsistent state. Where the system could not heal itself, engineers freed the agents and cleaned up job states by hand, which was finished by 11:56 UTC. Buildkite said the migration had exposed gaps in its review process.

Severity
–
Scope
Agent API severely degraded for most customers from 04:48 to 05:30 UTC and for some until 08:43 UTC; REST API, GraphQL and Web UI degraded from 04:48 to 08:43 UTC; some agents and jobs left inconsistent until 11:56 UTC
Users affected
Most Buildkite customers
Region
–
Time to detect
5m
Time to resolve
3h 55m

Symptoms

From 04:48 UTC on 10 November 2025, queries on the table that stores annotations in Buildkite Pipelines timed out, CPU load on the affected databases rose sharply, and the Agent API, the REST API, GraphQL and the Web UI slowed down and returned more errors.

Many of the timed-out queries came from the agent command that creates a new annotation. The status page first reported problems displaying build annotations, then increased error rates and slow performance on the REST and GraphQL APIs and the Web UI. The Agent API was severely degraded for most customers until 05:30 UTC and for some remaining customers until 08:43 UTC, and the REST API and GraphQL were degraded from 04:48 until 08:43 UTC.

Recreating the removed index restored about half of the database shards by 05:30 UTC. On the other shards the ongoing load from annotation queries kept the index from being rebuilt efficiently, so recovery there stalled until that load was cut off.

Symptom Intermittent timeoutsDatabase overloadHigh error rateHigh latency

Root cause

Root cause Faulty deployment

Component Database
Contributing factor Missed in reviewNo staged rollout
Trigger
At 04:48 UTC a database migration removed an index on the annotations table of the Pipelines product.

Detection

At 04:53 UTC the impact of the removed index was detected as queries timing out on the annotations table.

Remediation

Action items

Lessons

Buildkite said the incident was mainly caused by a database migration that exposed gaps in its review process.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.