IL-0091 UK Ministry of Justice · 2023-06-06kubernetesaws

Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together

The Cloud Platform team changed the instance type of the worker nodes with Terraform. The change recycled all nodes at the same time, so every pod on them was deleted at once and user services went down for a few minutes. At 10:30 the team found that the nodes were all being recycled together during the instance type change.

Users reported most services back up by 10:30, but the DPS service stayed unavailable until 11:21 because its users could not authenticate. The incident was declared at 11:00 and repaired at 11:21; the report gives 55 minutes as both the time to repair and the time to resolve. The team opened a ticket with AWS asking how to change the instance type without an outage and planned to write the procedure down as a runbook.

Severity: –
Scope: user services on the Cloud Platform, whose production pods were deleted at once, including the DPS service
Users affected: several Cloud Platform users reported outages
Region: –
Time to detect: –
Time to resolve: 55m

Symptoms

At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

By 10:30 users reported that their services were running again. At 10:50 a user reported that the DPS service was down because people could not authenticate into it; the DPS service was reported back up at 11:21.

Symptom Service outage

Root cause

Root cause Configuration change

Component Node pool Terraform

Contributing factor Excessive blast radius

Detection

Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

Remediation

–

Action items

Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type

Lessons

–

Sources

https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2023-06-06-user-services-down.html · runbook

Related cases

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
same root cause · shared Terraform, Node pool
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024
same root cause · shared Node pool
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
same root cause · shared Node pool
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023
same root cause · shared Node pool
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node pool, Terraform among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0091
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0091/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.