Incident Library
Applied filters ComponentPrometheusSymptomMonitoring outageOrganisationBuildkiteClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
    Buildkite2026
    Symptoms

    On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

    High latencyOOMKilledMonitoring outageAutoscaler misbehaviour
    Trigger

    Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

    Root cause
    Capacity exhaustion
    Component
    PrometheusContainer memory limitNode pool
    Contributing factor
    No resource requests or limits
    Remediation
    • Moved the load of the affected services back to AWS Fargate by following prepared documentation
    • Scaled up the services of a few customers with very large backlogs further by hand
    Action items
    • Upsize the underlying system nodes
    • Set higher requests and limits for the Prometheus server so it can handle more load
    • Review all new EKS resources and set any missing requests and limits
    • Add observability and monitors for EKS pod and node health
    View case
Check your cluster for this

This case involves Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container memory limitNode poolPrometheus
Contributing factor
No resource requests or limits
Organisation
Buildkite
  1. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
    Buildkite2026