Incident Library
Applied filters Root causeConfiguration changeFactorMissed in reviewOrganisationDFDSClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
    Symptoms

    After the switch to Traefik v2, functional tests indicated that everything was fine, and this was reported on Slack at 08:44 UTC. The first sign of trouble came at 09:26 UTC, when a developer reported a high count of HTTP 307 redirects, especially on the Okta service. At 09:32 UTC the team learned that the online booking system was down.

    Service outage
    Trigger

    At 08:22 UTC the AWS load balancers were updated to send requests to the new Traefik v2 reverse proxy as part of the migration away from Traefik v1.

    Root cause
    Configuration change
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Missed in reviewMonitoring blind spot
    Detection

    A developer reported a high count of HTTP 307 redirects, in particular on the Okta service, at 09:26 UTC, 64 minutes after the change; six minutes later the team was told the online booking system was down. The functional tests run after the switch had not caught it.

    Remediation
    • Rolled back the switch to Traefik v2, putting rollback ahead of investigation because of how critical the booking system is
    Action items
    • Consider how critical announcements can be made more visible
    • Consider whether the ALBs used for testing should have been shared with the teams so they could test their own scenarios
    Lessons

    Cloud Engineering cannot test every application scenario on its own, even after testing its own applications for weeks. A header that changes value behind a new ingress controller, here x-forwarded-proto going from https to http, can break an authentication flow while basic functional tests still pass. DFDS asks whether diffing request headers before and after the migration would have shown the problem in advance.

    View case
Check your cluster for this

This case involves Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Cloud load balancerIngress controller
Contributing factor
Monitoring blind spotMissed in review
Organisation
DFDS
  1. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021