Graph · Incident Library

Root cause
Component
Symptom
Contributing factor
Case

Cases1 case

1 to 1 of 1

IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
Buildkite2026
Symptoms
On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.
High latency OOMKilled Monitoring outage Autoscaler misbehaviour
Trigger
Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.
Root cause
Capacity exhaustion
Component
Prometheus Container memory limit Node pool
Contributing factor
No resource requests or limits
Remediation
Moved the load of the affected services back to AWS Fargate by following prepared documentation
Scaled up the services of a few customers with very large backlogs further by hand
Action items
Upsize the underlying system nodes
Set higher requests and limits for the Prometheus server so it can handle more load
Review all new EKS resources and set any missing requests and limits
Add observability and monitors for EKS pod and node health
View case

Check your cluster for this

This case involves Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component: Container memory limit Node pool Prometheus
Contributing factor: No resource requests or limits
Organisation: Buildkite

IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
Buildkite2026