Cases4 cases

1 to 4 of 4

1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
3. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region

   Skyscanner2021

   Symptoms

   On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.

   Service outageResources deleted

   Trigger

   A peer-reviewed change to a rarely changed root-level file of the cells templating configuration, intended as a no-op, that left out the {{ }} delimiters; it was submitted around 15:30 GMT, merged to master and rolled out automatically to every region at once.

   Root cause

   Configuration change

   Component

   Argo CDConfiguration templating

   Contributing factor

   Missed in reviewNo staged rolloutNo configuration testsUntested backupAutomation amplificationExcessive blast radius

   Remediation

   • restored the correct configuration for each cluster and let Argo CD reconcile the cluster back to it
   • restored services through the backup and restoration process
   • brought one region back first with the critical services and served all traffic from it from around 20:30 GMT
   • restored and verified the remaining regions and non-critical services over Thursday and Friday

   Action items

   • Avoid global configuration deploys, including rare change paths like this one
   • Treat templated configuration as code and add testing and linting
   • Plan and wargame more drastic failure scenarios in runbooks
   • Verify backup and restore processes, including access to the backups and the loss of a region
   • Refactor runbooks so they are clear and, where appropriate, idempotent for a stressed engineer in the early morning
   • Reconsider how much configuration is templated across regions, weighing configuration drift against automated global rollout

   Lessons

   Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.

   View case
4. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

   DFDS2021

   Symptoms

   At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

   Service outageHigh error rate

   Trigger

   The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

   Root cause

   Configuration change

   Component

   AWS VPC CNICloud load balancerTerraform

   Contributing factor

   Unverified assumptionNo configuration tests

   Detection

   Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

   Remediation

   • committed a fix that reverted the infrastructure-modules reference
   • re-triggered the pipeline after it failed on an AWS API timeout
   • let the pipeline reattach the target groups to the ALB

   Action items

   • Revert the ALB de-registration delay to the default of 300 seconds
   • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
   • Create an ingress test suite

   Lessons

   The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

   View case

By root cause2