Incident Library
Applied filters FactorUnpatched known bugClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
    Symptoms

    At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.

    High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage
    Root cause
    Software defect
    Component
    kubelet
    Contributing factor
    Unpatched known bug
    Detection

    Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.

    Remediation
    • restarted the kubelets on the stuck worker nodes, which did not last
    • restarted all worker nodes in a staggered order starting at about 18:00 UTC, which took about 1.5 hours
    Action items
    • Restart nodes on a rolling schedule to keep nodes and kubelets healthy until the fixes land upstream and reach the cloud provider, Microsoft Azure
    Lessons

    When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.

    View case
Check your cluster for this

These 9 cases involve Linux kernel and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Breaking change in an upgrade1 case

Component
Cloud load balancerkube-proxyLinux kernelNode operating system image
Contributing factor
Unpatched known bugNo staged rollout
Organisation
Altinity
  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025

Software defect5 cases

Component
Container CPU limit2kubelet2Linux kernel2Cloud load balancer1Elasticsearch or OpenSearch1Kubernetes API server1Node operating system image1Telemetry agent1
Contributing factor
Unpatched known bug5Monitoring blind spot2Unverified assumption1Unknown limit1
Organisation
AirMap1Buffer1Omio1PrometheusKube1Zalando1
  1. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch
    PrometheusKube2020
  2. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  4. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB
    Zalando2019
  5. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018

Cloud provider outage1 case

Component
Application databaseNode root diskPersistent volumes and stateful sets
Contributing factor
Unverified assumptionUnpatched known bugOverlapping failures
Organisation
Grafana Labs
  1. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

DNS or network fault1 case

Component
conntrack tableLinux kernelNode operating system imageAWS VPC CNI
Contributing factor
Unreviewed defaultsUnpatched known bug
Organisation
Prezi
  1. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019