Cases2 cases

1 to 2 of 2

1. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour

   DFDS2021

   Symptoms

   After the switch to Traefik v2, functional tests indicated that everything was fine, and this was reported on Slack at 08:44 UTC. The first sign of trouble came at 09:26 UTC, when a developer reported a high count of HTTP 307 redirects, especially on the Okta service. At 09:32 UTC the team learned that the online booking system was down.

   Service outage

   Trigger

   At 08:22 UTC the AWS load balancers were updated to send requests to the new Traefik v2 reverse proxy as part of the migration away from Traefik v1.

   Root cause

   Configuration change

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Missed in reviewMonitoring blind spot

   Detection

   A developer reported a high count of HTTP 307 redirects, in particular on the Okta service, at 09:26 UTC, 64 minutes after the change; six minutes later the team was told the online booking system was down. The functional tests run after the switch had not caught it.

   Remediation

   • Rolled back the switch to Traefik v2, putting rollback ahead of investigation because of how critical the booking system is

   Action items

   • Consider how critical announcements can be made more visible
   • Consider whether the ALBs used for testing should have been shared with the teams so they could test their own scenarios

   Lessons

   Cloud Engineering cannot test every application scenario on its own, even after testing its own applications for weeks. A header that changes value behind a new ingress controller, here x-forwarded-proto going from https to http, can break an authentication flow while basic functional tests still pass. DFDS asks whether diffing request headers before and after the migration would have shown the problem in advance.

   View case
2. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

   DFDS2021

   Symptoms

   At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

   Service outageHigh error rate

   Trigger

   The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

   Root cause

   Configuration change

   Component

   AWS VPC CNICloud load balancerTerraform

   Contributing factor

   Unverified assumptionNo configuration tests

   Detection

   Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

   Remediation

   • committed a fix that reverted the infrastructure-modules reference
   • re-triggered the pipeline after it failed on an AWS API timeout
   • let the pipeline reattach the target groups to the ALB

   Action items

   • Revert the ALB de-registration delay to the default of 300 seconds
   • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
   • Create an ingress test suite

   Lessons

   The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

   View case

By root cause1