Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

Symptoms

On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

The affected notifications were GitHub commit status, email, Slack and webhook notifications. Behind the delays, the Prometheus server on Buildkite's EKS cluster needed more memory than its host had, the host killed the process, and the Kubernetes control plane restarted it. While Prometheus was down, the metrics that drove application autoscaling were missing, so the services did not scale up. Soon after each restart Prometheus ran out of the host's memory again and the cycle repeated, from 15:42 to 17:33.

A few customers built up such a large backlog during the slow period that their services had to be scaled up further by hand.

Symptom High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

Root cause

Trigger

Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

Detection

–

Remediation

• Moved the load of the affected services back to AWS Fargate by following prepared documentation
• Scaled up the services of a few customers with very large backlogs further by hand

Action items

• Upsize the underlying system nodes
• Set higher requests and limits for the Prometheus server so it can handle more load
• Review all new EKS resources and set any missing requests and limits
• Add observability and monitors for EKS pod and node health

Lessons

–

Sources

• https://www.buildkitestatus.com/incidents/3vngjcnbkycl · official-status · archived copy

Related cases

• IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

  same root cause · shared Prometheus, Container memory limit, Node pool

• IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026

  same root cause · shared Container memory limit, Node pool

• IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025

  same root cause · shared Container memory limit, Node pool

• IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

  same root cause · shared Container memory limit, Node pool

• IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

  same root cause · shared Prometheus, Container memory limit

• IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

  same root cause · shared Prometheus, Node pool

• IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

  same root cause · shared Container memory limit, Node pool

• IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchangedUC Berkeley · 2017

  same root cause · shared Container memory limit, Node pool

• IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026

  same root cause · shared Node pool

• IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscaleBuildkite · 2026

  same root cause · shared Prometheus

• IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

  same root cause · shared Node pool

• IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025

  same root cause · shared Node pool

• IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024

  same root cause · shared Prometheus

• IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

  same root cause · shared Container memory limit

• IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

  same root cause · shared Container memory limit

• IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023

  same root cause · shared Prometheus

• IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023

  same root cause · shared Container memory limit

• IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

  same root cause · shared Container memory limit

• IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

  same root cause · shared Prometheus

• IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

  same root cause · shared Node pool

• IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

  same root cause · shared Node pool

• IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019

  same root cause · shared Container memory limit

• IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

  same root cause · shared Container memory limit

• IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-loopedBinder Project · 2017

  same root cause · shared Container memory limit

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0548
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0548/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.