Cases7 cases

1 to 5 of 7

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

   Kevin2026

   Symptoms

   An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

   CPU throttlingHigh latency

   Trigger

   A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitHorizontal pod autoscaler

   Contributing factor

   Monitoring blind spot

   Detection

   Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

   Remediation

   • raised the HPA minReplicas from 5 to 10
   • raised the container CPU limit from 1 core to 2 cores

   Action items

   • Keep adjusting the CPU limit to a suitable level through monitoring

   Lessons

   CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do. Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

   View case
3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case
4. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes

   GitLab2021

   Symptoms

   The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

   High latencyAutoscaler misbehaviour

   Trigger

   Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

   Root cause

   Configuration change

   Component

   Horizontal pod autoscaler

   Contributing factor

   Unverified assumption

   Detection

   The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

   Remediation

   • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Action items

   • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Lessons

   When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

   View case

By root cause5