Incident Library
Applied filters ComponentContainer memory limitSymptomMonitoring outageFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
    Symptoms

    At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

    Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage
    Trigger

    Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitkubelet
    Contributing factor
    Single point of failureRetry storm
    Detection

    Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

    Remediation
    • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
    • deployed additional replicas of key services
    • purged the registry cache and monitored the clusters as they stabilised
    • adjusted service configurations after the incident to keep the platform stable
    Action items
    • Implement a local authentication cache
    • Further develop node authentication
    • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols
    Lessons

    Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

    View case
  2. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
    Symptoms

    Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

    High error rateDNS resolution failuresOOMKilledMonitoring outage
    Trigger

    Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSContainer memory limitClient-side DNS resolver and cache
    Contributing factor
    Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot
    Detection

    The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

    Remediation
    • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi
    Action items
    • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
    • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
    • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot
    Lessons

    The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

    View case
Check your cluster for this

These 2 cases involve Container memory limit and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Container memory limit2CoreDNS1Client-side DNS resolver and cache1kubelet1
Contributing factor
Retry storm2Excessive blast radius1Unreviewed defaults1Unknown limit1Monitoring blind spot1No caching1Single point of failure1
Organisation
Scaleway1Zalando1
  1. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  2. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019