IL-0111 DFDS · 2021-08-26kubernetesawsnetworking
On August 26, 2021, as part of moving off Traefik v1, DFDS made Traefik v2 the main Kubernetes ingress controller by pointing the AWS load balancers at it. A misconfiguration kept Traefik v2 from forwarding the X-Forwarded headers, so x-forwarded-proto reached applications as http instead of https. The Okta authentication workflow broke for every service that used Okta, and the online booking system was completely down for a little over an hour.
Functional tests after the switch looked fine, and the problem was noticed 64 minutes later through a developer's report of many HTTP 307 redirects. Because the booking system was critical, the team rolled back before investigating further, and service was back at 09:49 UTC, 87 minutes after the change. The root cause was found afterwards in a deep dive with the Okta team. Among the contributing factors, DFDS lists the overlooked header change, the long detection time, the expectation that every team watches its own application after critical changes, and Slack announcements that get lost among other messages.
After the switch to Traefik v2, functional tests indicated that everything was fine, and this was reported on Slack at 08:44 UTC. The first sign of trouble came at 09:26 UTC, when a developer reported a high count of HTTP 307 redirects, especially on the Okta service. At 09:32 UTC the team learned that the online booking system was down.
The cause turned out to be request headers: Traefik v2 was configured so that it did not forward the insecure "X-FORWARDED-" headers, and the x-forwarded-proto header that applications received changed from https to http. That broke the Okta authentication workflow for every service using Okta. The headers had been checked before the migration, but this change was missed.
Symptom Service outage
Root cause Configuration change
A developer reported a high count of HTTP 307 redirects, in particular on the Okta service, at 09:26 UTC, 64 minutes after the change; six minutes later the team was told the online booking system was down. The functional tests run after the switch had not caught it.
Cloud Engineering cannot test every application scenario on its own, even after testing its own applications for weeks. A header that changes value behind a new ingress controller, here x-forwarded-proto going from https to http, can break an authentication flow while basic functional tests still pass. DFDS asks whether diffing request headers before and after the migration would have shown the problem in advance.
IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022same root cause · shared Cloud load balancer, Ingress controller
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Cloud load balancer
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022same root cause · shared Cloud load balancer
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021same root cause · shared Cloud load balancer
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
IL-0052same root cause · shared Cloud load balancer