Incident Library
Applied filters ComponentTerraformDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

8 to 9 of 9

  1. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
    Symptoms

    On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

    Service outage
    Trigger

    Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

    Root cause
    Configuration change
    Component
    TerraformCloud load balancerNode launch template
    Contributing factor
    Unrepresentative stagingSignal buried in noiseMonitoring blind spot
    Detection

    A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

    Remediation
    • Attached the worker node instances to the ALBs by hand
    • Implemented a permanent fix within the hour
    Action items
    • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
    • Reduce alert noise further
    • Implement host name resolution for Goldpinger
    Lessons

    General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

    View case
  2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
    Symptoms

    Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

    Resources deleted
    Trigger

    A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

    Root cause
    Configuration change
    Component
    TerraformCI systemGKE control plane
    Contributing factor
    No safeguard on destructive changeMissed in reviewUnverified assumption
    Detection

    Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

    Remediation
    • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
    • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
    • recreated the clusters, using the hourly backups
    • updated the places where the old clusters' master IPs were hardcoded
    Action items
    • Post the Terraform dry-run output as a comment on each pull request
    • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
    • Require an approving review on pull requests
    • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
    • Add linters and validators for the cluster configuration
    • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures
    Lessons

    Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

    View case
Check your cluster for this

These 9 cases involve Terraform and Node launch template. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Operator error2 cases

Component
Terraform2CI system1Ingress controller1Jobs and CronJobs1Persistent volumes and stateful sets1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change2No runbook1
Organisation
2i2c1UK Ministry of Justice1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024

Software defect2 cases

Component
Terraform2kubelet1Node launch template1Linux kernel1Node operating system image1Node pool1AWS VPC CNI1
Contributing factor
Automation amplification1Monitoring blind spot1No health probes1No staged rollout1Overlapping failures1Deferred upgrades1Unrepresentative staging1
Organisation
DFDS1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022

Configuration change5 cases

Component
Terraform5Cloud load balancer2Node launch template2Node pool2CI system1Container runtime1GKE control plane1Telemetry agent1AWS VPC CNI1
Contributing factor
Unverified assumption2No configuration tests2No runbook2Unrepresentative staging2Signal buried in noise1Automation amplification1Excessive blast radius1Unreviewed defaults1Harmful manual action1Monitoring blind spot1No safeguard on destructive change1Missed in review1
Organisation
DFDS3Spotify1UK Ministry of Justice1
  1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  3. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  4. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  5. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019