Cases2 cases

1 to 2 of 2

1. IL-0547 Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index

   Buildkite2025

   Symptoms

   From 04:48 UTC on 10 November 2025, queries on the table that stores annotations in Buildkite Pipelines timed out, CPU load on the affected databases rose sharply, and the Agent API, the REST API, GraphQL and the Web UI slowed down and returned more errors.

   Intermittent timeoutsDatabase overloadHigh error rateHigh latency

   Trigger

   At 04:48 UTC a database migration removed an index on the annotations table of the Pipelines product.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Missed in reviewNo staged rollout

   Detection

   At 04:53 UTC the impact of the removed index was detected as queries timing out on the annotations table.

   Remediation

   • Recreated the removed index, finishing on about half of the database shards by 05:30 UTC
   • Temporarily revoked the application user's permission to read from and write to the annotations table
   • Increased max_parallel_maintenance_workers in Postgres to speed up index creation
   • Finished recreating the index on the remaining shards between 07:50 and 08:43 UTC
   • Freed the agents and cleaned up the job states left inconsistent, handling by hand the cases the system could not self-heal, by 11:56 UTC

   Action items

   • Extend the internal tooling that flags risky migrations in code review to cover index removals and ask engineers to verify an index is no longer needed
   • Roll out high-risk migrations to shards in phases instead of to all shards at once
   • Evaluate tools such as HypoPG to analyse index creation and removal
   • Show index usage data from pganalyze more prominently in the development workflow
   • Automate the manual cleanup of inconsistent agent states so the system can heal itself

   Lessons

   Buildkite said the incident was mainly caused by a database migration that exposed gaps in its review process.

   View case
2. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour

   Open Collective2023

   Symptoms

   At 09:20 UTC on 4 April 2023, seven minutes after a deployment was triggered, Open Collective's service degraded: a migration in the deployment had locked the entire Transactions table in the production database.

   Lock contention or deadlockDatabase overload

   Trigger

   A deployment ran a migration that dropped indexes on the Transactions table and re-created them with CONCURRENTLY, excluding soft-deleted rows.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Missed in reviewUnrepresentative staging

   Detection

   The degradation of service was noticed at 09:20 UTC, seven minutes after the deployment was triggered.

   Remediation

   • Killed the migration, though the table remained locked
   • Put the frontend into maintenance mode to limit queries
   • Closed the other connections to the database
   • Cancelled the new index creation and restored the old index
   • Restored the frontend and confirmed the issue was resolved

   Action items

   • Add a script that simulates production load locally, using a tool like vegeta
   • Run this kind of migration during specific maintenance windows
   • Create the new index first and drop the old one afterwards in future migrations
   • Insist on a review from another engineer before merging such a pull request

   Lessons

   Dropping the old index before creating the new one was too optimistic under constant production queries, even with CONCURRENTLY, and an unreviewed pull request let it through. Tests locally and on staging did not show the problem because there were no tools to run migrations in stressed environments. Putting the frontend in maintenance mode limited the impact while showing users a clear message.

   View case

By root cause1