A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service

Symptoms

On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

The alert came during planned maintenance that set off many alerts, on top of alert noise that was already high, and DFDS had no alert on healthy or unhealthy ALB targets, only indirect alerts for the attached instances.

Symptom Service outage

Root cause

Trigger

Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

Detection

A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

Remediation

• Attached the worker node instances to the ALBs by hand
• Implemented a permanent fix within the hour

Action items

• Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
• Reduce alert noise further
• Implement host name resolution for Goldpinger

Lessons

General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

Sources

• https://github.com/dfds/postmortems/blob/master/PM2020-005%20-%20ALB%20instance%20attachment.md · github

Related cases

• IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

  same root cause · shared Node launch template, Terraform

• IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

  same root cause · shared Terraform, Cloud load balancer

• IL-0346 Render's routing service returned intermittent 404s for web services and static sites in OregonRender · 2025

  same root cause · shared Cloud load balancer

• IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutesTwingate · 2025

  same root cause · shared Cloud load balancer

• IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

  same root cause · shared Cloud load balancer

• IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutesRailway · 2024

  same root cause · shared Terraform

• IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

  same root cause · shared Node launch template

• IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate themJenkins project · 2023

  same root cause · shared Cloud load balancer

• IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

  same root cause · shared Terraform

• IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to themTwingate · 2023

  same root cause · shared Terraform

• IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023

  same root cause · shared Terraform

• IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listenerPython Software Foundation · 2022

  same root cause · shared Cloud load balancer

• IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE Plus · 2022

  same root cause · shared Cloud load balancer

• IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

  same root cause · shared Cloud load balancer

• IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

  same root cause · shared Cloud load balancer

• IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

  same root cause · shared Terraform

• IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

  same root cause · shared Cloud load balancer

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0365
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0365/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.