Incident Library
Applied filters ComponentNode poolOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

5 to 6 of 6

  1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
    Symptoms

    At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

    Service outage
    Trigger

    The team changed the instance type of the cluster's worker nodes through Terraform.

    Root cause
    Configuration change
    Component
    Node poolTerraform
    Contributing factor
    Excessive blast radiusNo runbook
    Detection

    Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

    Action items
    • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
    • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
    View case
  2. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
    Symptoms

    At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

    Image pull failures
    Trigger

    The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

    Root cause
    Operator error
    Component
    Container registry and its credentialsNode launch templateNode poolCI system
    Contributing factor
    Credential in configurationShared credential
    Detection

    A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

    Remediation
    • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
    • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again
    Action items
    • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
    • Rotate the password cleanly out of hours with a graceful recycle of the nodes
    Lessons

    The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

    View case
Check your cluster for this

These 6 cases involve Node pool and Node launch template. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Faulty deployment1 case

Component
Cluster autoscalerDeschedulerNode pool
Contributing factor
Unreviewed defaultsNo alert on limitNo headroom
Organisation
UK Ministry of Justice
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025

Configuration change3 cases

Component
Node pool3AWS VPC CNI2CoreDNS1Node launch template1Node root disk1Terraform1
Contributing factor
No configuration tests2Excessive blast radius1No alert on limit1No runbook1Missed in review1
Organisation
UK Ministry of Justice3
  1. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  2. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  3. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023

Capacity exhaustion1 case

Component
Container memory limitNode poolPrometheus
Contributing factor
No alert on limit
Organisation
UK Ministry of Justice
  1. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023

Operator error1 case

Component
CI systemNode launch templateNode poolContainer registry and its credentials
Contributing factor
Credential in configurationShared credential
Organisation
UK Ministry of Justice
  1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023