Cases3 cases

1 to 3 of 3

1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case
2. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

   Buildkite2026

   Symptoms

   On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

   High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

   Trigger

   Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limitNode pool

   Contributing factor

   No resource requests or limits

   Remediation

   • Moved the load of the affected services back to AWS Fargate by following prepared documentation
   • Scaled up the services of a few customers with very large backlogs further by hand

   Action items

   • Upsize the underlying system nodes
   • Set higher requests and limits for the Prometheus server so it can handle more load
   • Review all new EKS resources and set any missing requests and limits
   • Add observability and monitors for EKS pod and node health

   View case
3. IL-0547 Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index

   Buildkite2025

   Symptoms

   From 04:48 UTC on 10 November 2025, queries on the table that stores annotations in Buildkite Pipelines timed out, CPU load on the affected databases rose sharply, and the Agent API, the REST API, GraphQL and the Web UI slowed down and returned more errors.

   Intermittent timeoutsDatabase overloadHigh error rateHigh latency

   Trigger

   At 04:48 UTC a database migration removed an index on the annotations table of the Pipelines product.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Missed in reviewNo staged rollout

   Detection

   At 04:53 UTC the impact of the removed index was detected as queries timing out on the annotations table.

   Remediation

   • Recreated the removed index, finishing on about half of the database shards by 05:30 UTC
   • Temporarily revoked the application user's permission to read from and write to the annotations table
   • Increased max_parallel_maintenance_workers in Postgres to speed up index creation
   • Finished recreating the index on the remaining shards between 07:50 and 08:43 UTC
   • Freed the agents and cleaned up the job states left inconsistent, handling by hand the cases the system could not self-heal, by 11:56 UTC

   Action items

   • Extend the internal tooling that flags risky migrations in code review to cover index removals and ask engineers to verify an index is no longer needed
   • Roll out high-risk migrations to shards in phases instead of to all shards at once
   • Evaluate tools such as HypoPG to analyse index creation and removal
   • Show index usage data from pganalyze more prominently in the development workflow
   • Automate the manual cleanup of inconsistent agent states so the system can heal itself

   Lessons

   Buildkite said the incident was mainly caused by a database migration that exposed gaps in its review process.

   View case

By root cause2