Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
    Symptoms

    At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

    Service outage
    Trigger

    The team changed the instance type of the cluster's worker nodes through Terraform.

    Root cause
    Configuration change
    Component
    Node poolTerraform
    Contributing factor
    Excessive blast radiusNo runbook
    Detection

    Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

    Action items
    • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
    • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
    View case
Check your cluster for this

This case involves Node pool and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Node poolTerraform
Contributing factor
Excessive blast radiusNo runbook
Organisation
UK Ministry of Justice
  1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023