Incident Library

Applied filters ComponentTerraformFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases1 case

1 to 1 of 1

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
UK Ministry of Justice2023
Symptoms
At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.
Service outage
Trigger
The team changed the instance type of the cluster's worker nodes through Terraform.
Root cause
Configuration change
Component
Node pool Terraform
Contributing factor
Excessive blast radius No runbook
Detection
Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.
Action items
Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
View case

Check your cluster for this

This case involves Node pool and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component: Node pool Terraform
Contributing factor: Excessive blast radius No runbook
Organisation: UK Ministry of Justice

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
UK Ministry of Justice2023