Cases29 cases

9 to 13 of 29

1. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
2. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang

   Linode2025

   Symptoms

   On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

   DNS resolution failuresControl plane unresponsive

   Trigger

   An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and recordsDatabase

   Contributing factor

   –

   Detection

   The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

   Remediation

   • restarted the affected LKE DNS services

   Action items

   • Improve how the DNS services recover from transient connectivity failures to their dependencies
   • Improve monitoring to detect similar problems sooner
   • Update the deployment process to better coordinate changes between dependent systems

   Lessons

   A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

   View case
3. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
4. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
5. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets

   Cybozu2024

   Symptoms

   A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

   DNS resolution failuresIntermittent timeoutsPacket loss

   Trigger

   The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

   Root cause

   Capacity exhaustion

   Component

   Linux kernelClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

   Remediation

   • Raised the kernel parameter netdev_max_backlog on the nodes
   • Set up alerts on softnet
   • Wrote a runbook for the softnet alerts

   Action items

   • Evaluate Cilium's netkit device so pod packets bypass the backlog queue

   Lessons

   Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

   View case

By root cause6