Incident Library
Applied filters FactorChange with a delayed effectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

10 to 11 of 11

  1. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
    Symptoms

    A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

    High error rateThroughput degradationDeploys failing
    Trigger

    A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

    Root cause
    Software defect
    Component
    Admission webhookHorizontal pod autoscaler
    Contributing factor
    Change with a delayed effectMonitoring blind spotIgnored early signal
    Detection

    The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

    Remediation
    • deleted the mutating admission controller from the affected cluster
    Action items
    • Create alerts from the existing metrics that would have shown the problem earlier
    Lessons

    Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

    View case
  2. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 11 cases involve Ingress controller and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Breaking change in an upgrade3 cases

Component
Ingress controller2cert-manager1Cilium1Cloud load balancer1Node operating system image1systemd-networkd1
Contributing factor
Change with a delayed effect3Automation amplification1Unreviewed defaults1Monitoring blind spot1No staged rollout1Unrepresentative staging1
Organisation
2i2c1Datadog1Honeycomb1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  3. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023

Software defect4 cases

Component
Admission webhook1Automated capacity manager1Cloud load balancer1GPU driver1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1
Contributing factor
Change with a delayed effect4Monitoring blind spot3Ignored early signal2Automation amplification1Excessive blast radius1No configuration tests1No safeguard on destructive change1Single point of failure1
Organisation
Airbnb1Barush Mendez1DFDS1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  3. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  4. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Configuration change3 cases

Component
Cloud load balancer1ConfigMaps and Secrets1conntrack table1HAProxy1Ingress controller1kubelet1Linux kernel1Node operating system image1
Contributing factor
Change with a delayed effect3Unverified assumption2Excessive blast radius1Unreviewed defaults1Limited debug access1Retry storm1
Organisation
DigitalOcean1LINE+1Pusher1
  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  3. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019