Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

10 to 11 of 11

  1. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
    Symptoms

    On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

    High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
    Trigger

    Several services sent a high volume of external DNS queries, all of which went through kube-dns.

    Root cause
    Capacity exhaustion
    Component
    kube-dnskube-proxyClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

    Lessons

    By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

    View case
  2. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
    Symptoms

    After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

    Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures
    Trigger

    Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

    Root cause
    Capacity exhaustion
    Component
    Kubernetes API serverTelemetry agentetcd
    Contributing factor
    Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning
    Detection

    Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

    Remediation
    • moved internal users back to the old Bocce infrastructure
    • resized the three master nodes from m4.large to m4.4xlarge
    • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
    • raised max-requests-inflight for non-mutating requests from 400 to 1200
    • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
    • switched client scripts from the ClusterFirst DNS policy to Default
    Action items
    • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
    • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster
    Lessons

    Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

    View case
Check your cluster for this

These 11 cases involve CoreDNS and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion11 cases

Component
CoreDNS3Application database3Node pool3In-memory cache2Container CPU limit2Client-side DNS resolver and cache2Horizontal pod autoscaler2Container memory limit2Pod scheduling and placement2etcd1GKE control plane1Karpenter1Kubernetes API server1kube-dns1kube-proxy1kubelet1Language runtime settings1Prometheus1Service discovery1Telemetry agent1
Contributing factor
Unreviewed defaults3Monitoring blind spot3Retry storm3Automation amplification2Single point of failure2Unverified assumption1Change during peak hours1Unknown limit1Harmful manual action1No alert on limit1No caching1No capacity planning1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Airbnb1Backpackr1Civis Analytics1GitLab1Google Cloud1Honeycomb1Kevin1PostHog1RebelMouse1Temporal1이언철1
  1. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  4. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  5. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  6. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  7. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  8. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  9. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  10. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  11. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019