Cases1 case

1 to 1 of 1

1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case

By root cause1