IL-0470 Buildkite · 2026-05-28kubernetesawsobservability
Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale
Buildkite was moving its notification-processing workloads from its earlier ECS-based infrastructure to EKS. On 28 May 2026 the Prometheus service used by the EKS autoscaling path ran out of storage, so some EKS-based workers could not autoscale correctly while queues grew. From 20:01 UTC the notification-processing workers were underprovisioned and notification delivery latency rose. Internal queue latency monitoring caught it and engineers began moving the affected workloads from EKS back to ECS. The rollback took longer than expected because the ECS services were not ready to take the full load at once, and engineers had to adjust scaling configuration and worker counts by hand during the incident. Latency recovered for most customers by 21:00 UTC.
Some workloads were still running on EKS and had started autoscaling again after Prometheus recovered, but the team believed they were no longer serving traffic. When the infrastructure configuration was reconciled, those EKS workloads were scaled down before their ECS equivalents had fully scaled up, and from 22:12 to 22:40 UTC some notification-processing workers were underprovisioned again for a subset of customers. Completing the rollback and scaling the remaining affected ECS services ended it. Buildkite also increased Prometheus storage, added an alert ahead of storage exhaustion, and fixed GitHub notification retry behaviour for a class of errors that could cause repeated retries and lengthen notification delays. The public status page posted its first investigating notice at 20:20 UTC and marked the incident resolved at 21:18 UTC.
- Severity
- –
- Scope
- Delayed Buildkite notifications (GitHub commit status, email, Slack and webhook) from 20:01 to about 21:00 UTC, and again from 22:12 to 22:40 UTC for a subset of customers; downstream CI, merge and deployment workflows that depend on the notifications were delayed
- Users affected
- A subset of Buildkite customers
- Region
- –
- Time to detect
- –
- Time to resolve
- 2h 39m
Symptoms
From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.
For customers who feed Buildkite notifications into other CI, merge or deployment systems, the delay also held up those workflows, and some customers saw secondary or longer effects depending on the notification type, retry behaviour or integration involved. Notification latency recovered for most customers by 21:00 UTC.
A second, shorter delay hit a subset of customers from 22:12 to 22:40 UTC, after the first recovery, when EKS workloads that were still serving traffic were scaled down before their ECS replacements had fully scaled up.
Root cause
Root cause Capacity exhaustion
- Trigger
- The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.
Detection
Internal queue latency monitoring detected the rising notification delivery latency.
Remediation
- Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
- Adjusted scaling configuration and worker counts by hand
- Completed the rollback and scaled the remaining affected ECS services
- Increased Prometheus storage capacity and recorded the change in infrastructure as code
Action items
- Alert before Prometheus storage exhaustion can affect autoscaling
- Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
- Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
- Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
- Move customer-critical workloads to EKS more gradually, with clearer settling periods
- Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
- Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
- Update the status page earlier when notification latency is likely to affect customer workflows
- Describe the customer-visible impact in status page updates, not only the affected internal service
- Improve internal escalation for customers who report critical CI impact
Lessons
Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.
Sources
- https://www.buildkitestatus.com/incidents/8zy2b81jz4r9 · official-status · archived copy
Related cases
IL-0548Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs laggedBuildkite · 2026same root cause · shared Prometheus
IL-0044Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Prometheus
IL-0046Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024same root cause · shared Prometheus
IL-0016Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Prometheus
IL-0028Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Prometheus
IL-0070Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Prometheus
IL-0176Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared Prometheus
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0470
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0470/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.