Incident Library
Applied filters Root causeCapacity exhaustionSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases12 cases

11 to 12 of 12

  1. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
    Symptoms

    Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

    High error rateDNS resolution failuresOOMKilledMonitoring outage
    Trigger

    Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSContainer memory limitClient-side DNS resolver and cache
    Contributing factor
    Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot
    Detection

    The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

    Remediation
    • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi
    Action items
    • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
    • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
    • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot
    Lessons

    The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

    View case
  2. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
    Symptoms

    Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.

    High error rateControl plane unresponsiveOOMKilled
    Trigger

    A CronJob (apiVersion batch/v2alpha1, schedule "*/15 9-19 * * Mon-Fri") whose concurrencyPolicy: Forbid and job history limits sat inside the pod template spec instead of the CronJob spec. The talk shows it right after the slides on the number of pods and the API server being OOM-killed, and the lessons add a quota on the number of pods.

    Root cause
    Capacity exhaustion
    Component
    Kubernetes API serverIngress controller
    Contributing factor
    Single point of failureExcessive blast radius
    Remediation
    • fixed the CronJob manifest: moved concurrencyPolicy: Forbid and the job history limits to the CronJob spec, added activeDeadlineSeconds: 120 and changed the schedule to "7 8-18 * * Mon-Fri"
    Action items
    • Make the ingress stay healthy during API server problems
    • Make the ingress keep the last known set of routes
    • Use a ResourceQuota on the number of pods (the example sets pods: "1500")
    Lessons

    An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.

    View case
Check your cluster for this

These 12 cases involve Application database and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion12 cases

Component
Application database4Kubernetes API server4CoreDNS3Node pool3In-memory cache2etcd2Container memory limit2Pod scheduling and placement2CI system1conntrack table1Credential broker for pods1Client-side DNS resolver and cache1GKE control plane1HAProxy1Ingress controller1kube-proxy1kubelet1Linux kernel1Service discovery1Telemetry agent1
Contributing factor
Retry storm4Single point of failure4Unreviewed defaults3Unknown limit3Unverified assumption2Automation amplification2Excessive blast radius2Monitoring blind spot2No alert on limit2No capacity planning2Change during peak hours1Harmful manual action1No caching1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Zalando3Civis Analytics1GitLab1Google Cloud1Honeycomb1loveholidays1Moonlight1PostHog1RebelMouse1Render1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  3. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  4. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  5. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  6. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  7. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  8. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  9. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  10. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  11. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  12. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018