Cases7 cases

6 to 7 of 7

1. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region

   Skyscanner2021

   Symptoms

   On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.

   Service outageResources deleted

   Trigger

   A peer-reviewed change to a rarely changed root-level file of the cells templating configuration, intended as a no-op, that left out the {{ }} delimiters; it was submitted around 15:30 GMT, merged to master and rolled out automatically to every region at once.

   Root cause

   Configuration change

   Component

   Argo CDConfiguration templating

   Contributing factor

   Missed in reviewNo staged rolloutNo configuration testsUntested backupAutomation amplificationExcessive blast radius

   Remediation

   • restored the correct configuration for each cluster and let Argo CD reconcile the cluster back to it
   • restored services through the backup and restoration process
   • brought one region back first with the critical services and served all traffic from it from around 20:30 GMT
   • restored and verified the remaining regions and non-critical services over Thursday and Friday

   Action items

   • Avoid global configuration deploys, including rare change paths like this one
   • Treat templated configuration as code and add testing and linting
   • Plan and wargame more drastic failure scenarios in runbooks
   • Verify backup and restore processes, including access to the backups and the loss of a region
   • Refactor runbooks so they are clear and, where appropriate, idempotent for a stressed engineer in the early morning
   • Reconsider how much configuration is templated across regions, weighing configuration drift against automated global rollout

   Lessons

   Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.

   View case
2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one

   Spotify2019

   Symptoms

   Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

   Resources deleted

   Trigger

   A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

   Root cause

   Configuration change

   Component

   TerraformCI systemGKE control plane

   Contributing factor

   No safeguard on destructive changeMissed in reviewUnverified assumption

   Detection

   Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

   Remediation

   • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
   • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
   • recreated the clusters, using the hourly backups
   • updated the places where the old clusters' master IPs were hardcoded

   Action items

   • Post the Terraform dry-run output as a comment on each pull request
   • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
   • Require an approving review on pull requests
   • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
   • Add linters and validators for the cluster configuration
   • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures

   Lessons

   Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

   View case

By root cause1