Incident Library
Applied filters FactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases14 cases

14 to 14 of 14

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 14 cases involve Kubernetes API server and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Software defect4 cases

Component
Admission webhook1Configuration templating1GPU driver1HAProxy1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1Model request routing1TPU1XLA compiler1
Contributing factor
Ignored early signal4Monitoring blind spot3Change with a delayed effect2Unverified assumption1Limited debug access1Overlapping failures1Single point of failure1Unrepresentative staging1
Organisation
Airbnb1Anthropic1Barush Mendez1Skyscanner1
  1. IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responses
    Anthropic2025
  2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  3. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  4. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019

DNS or network fault2 cases

Component
CoreDNS2conntrack table1Client-side DNS resolver and cache1kube-proxy1Telemetry agent1
Contributing factor
Ignored early signal2Unverified assumption1Automation amplification1Unreviewed defaults1Unknown limit1No caching1
Organisation
Adevinta1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020

Configuration change2 cases

Component
Cloud load balancer1Cluster dashboard1Configuration templating1Container memory limit1Service discovery1
Contributing factor
Unreviewed defaults2Ignored early signal2Excessive blast radius1Monitoring blind spot1Unrepresentative staging1
Organisation
Grafana Labs1JW Player1
  1. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  2. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019

Certificate or credential expiry1 case

Component
Admission webhookPolicy engine
Contributing factor
Unreviewed defaultsIgnored early signalMonitoring blind spotDeferred upgrades
Organisation
DFDS
  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022

Capacity exhaustion3 cases

Component
Container runtime1etcd1Helm1Kubernetes API server1kubelet1Container memory limit1Node root disk1Cluster authentication provider1Telemetry agent1
Contributing factor
Ignored early signal3Unreviewed defaults2Limited debug access1Harmful manual action1Monitoring blind spot1No resource requests or limits1Retry storm1Single point of failure1
Organisation
Blue Matador1Fullstaq1Saltside1
  1. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  2. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  3. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Cloud provider outage1 case

Component
Cluster autoscalerGKE control planeHorizontal pod autoscalerIngress controllerNode pool
Contributing factor
Ignored early signalOverlapping failuresUnrepresentative staging
Organisation
loveholidays
  1. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018