IL-0105 DFDS · 2021-06-22kubernetesawsnetworking
DFDS was refactoring and cleaning up its infrastructure code so that EKS add-ons would be managed by native Terraform instead of kubectl. As part of that work the pipeline replaced the VPC CNI at 08:21 UTC. The change to the VPC CNI detached the target groups from the Application Load Balancer, so all inbound traffic failed to reach the cluster and every service answered 503.
The postmortem names two contributing factors: the de-registration delay of the ALB target groups had earlier been lowered permanently to speed up node roll-overs, and an essential network component was replaced. The team committed a fix at 08:33 that reverted the infrastructure-modules reference, re-ran the pipeline after an AWS API timeout, and at 08:46 the pipeline reattached the target groups to the ALB. Time to detection was 8 minutes and time to recovery 25 minutes.
At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.
Eight minutes before the alerts, at 08:21, the deployment pipeline had replaced the VPC CNI. The investigation started at 08:30, and when the fix was pushed through the pipeline it first failed on an AWS API timeout and had to be started again at 08:38.
Symptom Service outageHigh error rate
Root cause Configuration change
Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.
The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Cloud load balancer
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024same root cause · shared AWS VPC CNI
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023same root cause · shared AWS VPC CNI
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023same root cause · shared AWS VPC CNI
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Terraform
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1same root cause · shared Terraform
IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022same root cause · shared Cloud load balancer
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022same root cause · shared Cloud load balancer
IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021same root cause · shared Cloud load balancer
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019same root cause · shared Terraform
IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019same root cause · shared Cloud load balancer