Cases2 cases

1 to 2 of 2

1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
2. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region

   Skyscanner2021

   Symptoms

   On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.

   Service outageResources deleted

   Trigger

   A peer-reviewed change to a rarely changed root-level file of the cells templating configuration, intended as a no-op, that left out the {{ }} delimiters; it was submitted around 15:30 GMT, merged to master and rolled out automatically to every region at once.

   Root cause

   Configuration change

   Component

   Argo CDConfiguration templating

   Contributing factor

   Missed in reviewNo staged rolloutNo configuration testsUntested backupAutomation amplificationExcessive blast radius

   Remediation

   • restored the correct configuration for each cluster and let Argo CD reconcile the cluster back to it
   • restored services through the backup and restoration process
   • brought one region back first with the critical services and served all traffic from it from around 20:30 GMT
   • restored and verified the remaining regions and non-critical services over Thursday and Friday

   Action items

   • Avoid global configuration deploys, including rare change paths like this one
   • Treat templated configuration as code and add testing and linting
   • Plan and wargame more drastic failure scenarios in runbooks
   • Verify backup and restore processes, including access to the backups and the loss of a region
   • Refactor runbooks so they are clear and, where appropriate, idempotent for a stressed engineer in the early morning
   • Reconsider how much configuration is templated across regions, weighing configuration drift against automated global rollout

   Lessons

   Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.

   View case

By root cause1