Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 9 cases involve Cloud load balancer and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change9 cases

Component
Cloud load balancer3Terraform2Admission webhook1In-memory cache1CI system1Configuration templating1ConfigMaps and Secrets1conntrack table1Container runtime1Application database1GKE control plane1HAProxy1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Application logging library1Container memory limit1Persistent volumes and stateful sets1Telemetry agent1AWS VPC CNI1
Contributing factor
Unverified assumption9Excessive blast radius3Change with a delayed effect2No configuration tests2No safeguard on destructive change2Retry storm2Monitoring blind spot1No health probes1No headroom1No staged rollout1Missed in review1Single point of failure1
Organisation
DFDS2AdGuard VPN1GitLab1Grafana Labs1LINE+1MUSINSA1Pusher1Spotify1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  3. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  4. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  7. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  8. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  9. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019