IL-0244 UK Ministry of Justice · 2024-09-20kubernetesawsnetworking

A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

The Cloud Platform team had paused its infrastructure pipelines for a long time to make required manual updates to Terraform remote state. During that time a pull request that the team had not noticed was queued. When the infrastructure-vpc-live-1 pipeline was unpaused at 11:21, the queued pull request ran automatically and at 11:22 destroyed the route table associations of the EKS subnets, which removed internet routing to Cloud Platform services. The team ran terraform apply locally at 11:32 to rebuild the associations, told users at 11:33 that services were available again, and declared the incident resolved at 11:40.

Severity: –
Scope: Cloud Platform services on the live cluster were unavailable for a period of time
Users affected: –
Region: –
Time to detect: 2m
Time to resolve: 11m

Symptoms

On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

Symptom Service outage Network unreachable Resources deleted

Root cause

Root cause Operator error

Component Terraform CI system VPC route tables and firewall rules

Contributing factor No safeguard on destructive change

Trigger: At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

Detection

High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

Remediation

Ran terraform apply locally to rebuild the subnet route table associations

Action items

Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
Find a way to hold queued pull requests while manual infrastructure updates are under way
Find ways to isolate the plan and apply tasks of infrastructure pipelines better

Lessons

–

Sources

https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2024-09-20-eks-subnet-route-table.html · runbook · archived copy

Related cases

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025
same root cause · shared Terraform
IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023
same root cause · shared CI system
IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache serversAcquia · 2019
same root cause · shared VPC route tables and firewall rules

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Terraform, CI system, VPC route tables and firewall rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0244
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0244/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.