Incident Library
Applied filters Root causeConfiguration changeFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

7 to 9 of 9

  1. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
    Symptoms

    At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

    Deploys failingControl plane lag
    Trigger

    During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

    Root cause
    Configuration change
    Component
    Admission webhookKubernetes API server
    Contributing factor
    No configuration testsUnverified assumptionMonitoring blind spot
    Detection

    Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

    Remediation
    • toggled the Alcide admission controller to verify that it caused the timeouts
    • permanently deleted the Alcide admission controller
    Action items
    • Monitor Kubernetes deployments for timeouts
    • Monitor 5xx status codes from the Kubernetes API server
    Lessons

    New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

    View case
  2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
    Symptoms

    Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

    Resources deleted
    Trigger

    A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

    Root cause
    Configuration change
    Component
    TerraformCI systemGKE control plane
    Contributing factor
    No safeguard on destructive changeMissed in reviewUnverified assumption
    Detection

    Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

    Remediation
    • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
    • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
    • recreated the clusters, using the hourly backups
    • updated the places where the old clusters' master IPs were hardcoded
    Action items
    • Post the Terraform dry-run output as a comment on each pull request
    • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
    • Require an approving review on pull requests
    • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
    • Add linters and validators for the cluster configuration
    • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures
    Lessons

    Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

    View case
  3. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 9 cases involve Cloud load balancer and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change9 cases

Component
Cloud load balancer3Terraform2Admission webhook1In-memory cache1CI system1Configuration templating1ConfigMaps and Secrets1conntrack table1Container runtime1Application database1GKE control plane1HAProxy1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Application logging library1Container memory limit1Persistent volumes and stateful sets1Telemetry agent1AWS VPC CNI1
Contributing factor
Unverified assumption9Excessive blast radius3Change with a delayed effect2No configuration tests2No safeguard on destructive change2Retry storm2Monitoring blind spot1No health probes1No headroom1No staged rollout1Missed in review1Single point of failure1
Organisation
DFDS2AdGuard VPN1GitLab1Grafana Labs1LINE+1MUSINSA1Pusher1Spotify1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  3. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  4. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  7. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  8. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  9. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019