Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

11 to 11 of 11

  1. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 11 cases involve Ingress controller and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Breaking change in an upgrade3 cases

Component
Ingress controller2cert-manager1Cilium1Cloud load balancer1Node operating system image1systemd-networkd1
Contributing factor
Change with a delayed effect3Automation amplification1Unreviewed defaults1Monitoring blind spot1No staged rollout1Unrepresentative staging1
Organisation
2i2c1Datadog1Honeycomb1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  3. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023

Software defect4 cases

Component
Admission webhook1Automated capacity manager1Cloud load balancer1GPU driver1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1
Contributing factor
Change with a delayed effect4Monitoring blind spot3Ignored early signal2Automation amplification1Excessive blast radius1No configuration tests1No safeguard on destructive change1Single point of failure1
Organisation
Airbnb1Barush Mendez1DFDS1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  3. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  4. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Configuration change3 cases

Component
Cloud load balancer1ConfigMaps and Secrets1conntrack table1HAProxy1Ingress controller1kubelet1Linux kernel1Node operating system image1
Contributing factor
Change with a delayed effect3Unverified assumption2Excessive blast radius1Unreviewed defaults1Limited debug access1Retry storm1
Organisation
DigitalOcean1LINE+1Pusher1
  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  3. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019