Incident Library

Organisation

Buildkite

Buildkite · buildkite

1 case · 2026

Root cause Capacity exhaustion
Component CoreDNSHorizontal pod autoscalerContainer memory limitNode pool
Contributing factor Automation amplificationMonitoring blind spotNo headroomRetry storm
Symptom Autoscaler misbehaviourDNS resolution failuresOOMKilledService outage

Cases

  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026

    Root cause Capacity exhaustion