Incident Library
Applied filters SymptomService outageFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

17 to 17 of 17

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 17 cases involve Kubernetes API server and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Capacity exhaustion4 cases

Component
Cloud controller manager1conntrack table1Container runtime1CoreDNS1Helm1Horizontal pod autoscaler1Ingress controller1Istio1kubelet1Container memory limit1Node root disk1Node pool1Persistent volumes and stateful sets1
Contributing factor
Monitoring blind spot4Single point of failure3Unreviewed defaults2Retry storm2Automation amplification1Ignored early signal1Unknown limit1Limited debug access1No runbook1No headroom1
Organisation
Backpackr1Buildkite1Magic Pages1Saltside1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  3. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  4. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Dependency failure1 case

Component
Cloud block storageData center power and coolingetcdInfiniBand fabric
Contributing factor
Monitoring blind spotNo alert on limitNo runbookOverlapping failuresSingle point of failure
Organisation
Nebius
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026

Faulty deployment2 cases

Component
Ingress controller2Calico1Cloud load balancer1Pod termination1
Contributing factor
Monitoring blind spot2Unverified assumption1Excessive blast radius1Unreviewed defaults1No runbook1
Organisation
CircleCI1Yonghyun Kim1
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021

Software defect2 cases

Component
Kubernetes API server2etcd1GPU driver1kubelet1Linkerd1Linux kernel1
Contributing factor
Monitoring blind spot2Single point of failure2Change with a delayed effect1Ignored early signal1Harmful manual action1Overlapping failures1Ineffective rollback1
Organisation
Barush Mendez1Monzo1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Configuration change5 cases

Component
Kubernetes API server2Admission webhook1Argo CD1Cloud load balancer1CNI plugin1Configuration templating1Client-side DNS resolver and cache1GKE control plane1Ingress controller1kubelet1Persistent volumes and stateful sets1Telemetry agent1
Contributing factor
Monitoring blind spot5Automation amplification2Missed in review2Unrepresentative staging2Excessive blast radius1Cache-masked failure1Limited debug access1No health probes1No runbook1No safeguard on destructive change1No staged rollout1
Organisation
OpenAI2DFDS1InfluxData1Jetstack1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  4. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  5. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019

Breaking change in an upgrade2 cases

Component
Admission webhook1Calico1Cloud load balancer1etcd1Ingress controller1Kubernetes API server1
Contributing factor
Monitoring blind spot2Unrepresentative staging2Untested backup1Configuration not in code1Change with a delayed effect1Single point of failure1
Organisation
Honeycomb1Reddit1
  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018