Cases13 cases

1 to 5 of 13

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart

   BC Government2025sev1

   Symptoms

   The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

   Service outageContainer crash loopPersistent volume full

   Trigger

   A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsIn-memory cache

   Contributing factor

   No capacity planning

   Remediation

   • backed off the Redis StatefulSet and the deployment
   • deleted the corrupted PVC and rebuilt it
   • restarted the StatefulSet and the deployment

   Action items

   • Increase the Klamm Redis cache volume to 100Mi
   • Integrate Sysdig monitoring on the namespace
   • Enable replica pods on the Redis cluster

   View case
3. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution

   Mercari2025sev1

   Symptoms

   During peak hours, when Mercari's GKE clusters handle very high request rates, the platform team saw kube-dns reaching its limits. DNS query rates were high enough to overwhelm the kube-dns service.

   DNS resolution failuresIntermittent timeoutsService outage

   Trigger

   Peak-hour traffic on the GKE clusters produced more DNS queries than the few kube-dns pods could answer.

   Root cause

   Capacity exhaustion

   Component

   kube-dns

   Contributing factor

   No cachingSingle point of failure

   Detection

   During peak hours the team saw frequent NXDOMAIN and truncated-response errors and recurring "failed to refresh DNS cache" errors on the clusters, which showed that kube-dns had reached its limits.

   Remediation

   • deployed NodeLocal DNSCache on the Laboratory GKE cluster with a specific nodeAffinity to measure its effect
   • rolled NodeLocal DNSCache out gradually to the Dev and Prod environments by labelling node pools so its pods could be deployed there

   Lessons

   When every pod in a busy cluster resolves names through a few central kube-dns pods, DNS becomes a shared bottleneck, and a DNS failure quickly turns into higher latency, timeouts and cascading outages across microservices. A cache on each node takes most of that load off kube-dns and keeps resolution local to the node. Rolling the cache out first to an internal test cluster with nodeAffinity, and then node pool by node pool through labels, kept the blast radius of the change small.

   View case
4. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse

   RebelMouse2025

   Symptoms

   At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.

   DNS resolution failuresHigh latencyHigh error rateService outage

   Trigger

   At 13:15 UTC RebelMouse started its regular reroll of EKS nodes to apply security patches. The planned, automatic termination of an EC2 instance terminated one of the CoreDNS pods.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Change during peak hours

   Detection

   Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.

   Remediation

   • rescheduled the affected CoreDNS pod onto a new EC2 instance
   • scaled up the CoreDNS deployment to more replicas so DNS load was spread evenly across all pods

   Action items

   • Add scale-in protection so CoreDNS pods are not terminated unexpectedly during peak load
   • Review the CoreDNS configuration so that vertical scaling of the pods works
   • Hold regular capacity reviews so the CoreDNS deployment is provisioned for peak load

   View case
5. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case

By root cause1