IL-0106 DFDS · 2021-09-06kubernetesawsnetworking
DFDS routed ingress traffic for its Kubernetes services through Traefik v2 behind an AWS ALB. On 2021-09-01 a malformed IngressRoute manifest was applied, and the official Traefik v2 API schema did not stop it. Traefik v2 versions before 2.5 have a bug (Traefik issue 8098) that makes such a resource stop Traefik from receiving or routing any ingress traffic, but the bug only takes effect once all Traefik v2 pods have restarted. That happened during the automated node rollover on 2021-09-06, and from 08:00 UTC every Kubernetes service returned 404.
Because Traefik v2 failed gracefully, the team first chased the unhealthy ALB targets and changed the health check path, which did not bring traffic back. At 08:50 the team began rolling back to Traefik v1 as the main ingress controller, carried out the rollback by hand at 09:00, and ingress traffic worked again at 09:08. At 09:10 the team found the root cause by deleting the malformed IngressRoute by hand and confirmed the Traefik v2 bug, and at 09:20 the changes were applied upstream. Detection took 5 minutes and recovery 68 minutes.
At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.
At 08:10 the targets of the AWS Application Load Balancer (ALB) in front of the cluster were unhealthy. Changing the ALB health check path made the targets healthy again at 08:15, but at 08:20 all services were still returning 404. Traefik v2, the ingress controller, failed gracefully, which pointed the investigation at the ALB, and other checks up to 08:30 found nothing conclusive.
The resource behind the failure had been applied five days earlier: a malformed IngressRoute went into the cluster at 11:34 UTC on 2021-09-01. The failure appeared only after the automated node rollover that started at 07:30 on 2021-09-06 had restarted all the Traefik v2 pods.
Root cause Software defect
All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.
A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik.
On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Cloud load balancer
IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025same root cause · shared Ingress controller
IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025same root cause · shared Ingress controller
IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024same root cause · shared Cloud load balancer
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
IL-0062same root cause · shared Ingress controller
IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019same root cause · shared Ingress controller
IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019same root cause · shared Cloud load balancer
IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Ingress controller