Cases2 cases

1 to 2 of 2

1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

   UK Ministry of Justice2024

   Symptoms

   On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

   Service outageNetwork unreachableResources deleted

   Trigger

   At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

   Root cause

   Operator error

   Component

   TerraformCI systemVPC route tables and firewall rules

   Contributing factor

   No safeguard on destructive change

   Detection

   High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

   Remediation

   • Ran terraform apply locally to rebuild the subnet route table associations

   Action items

   • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
   • Find a way to hold queued pull requests while manual infrastructure updates are under way
   • Find ways to isolate the plan and apply tasks of infrastructure pipelines better

   View case
2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one

   Spotify2019

   Symptoms

   Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

   Resources deleted

   Trigger

   A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

   Root cause

   Configuration change

   Component

   TerraformCI systemGKE control plane

   Contributing factor

   No safeguard on destructive changeMissed in reviewUnverified assumption

   Detection

   Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

   Remediation

   • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
   • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
   • recreated the clusters, using the hourly backups
   • updated the places where the old clusters' master IPs were hardcoded

   Action items

   • Post the Terraform dry-run output as a comment on each pull request
   • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
   • Require an approving review on pull requests
   • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
   • Add linters and validators for the cluster configuration
   • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures

   Lessons

   Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

   View case

By root cause2