Cases3 cases

1 to 3 of 3

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
3. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case

By root cause1