Cases2 cases

1 to 2 of 2

1. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
2. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

   UK Ministry of Justice2023

   Symptoms

   At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

   Image pull failures

   Trigger

   The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

   Root cause

   Operator error

   Component

   Container registry and its credentialsNode launch templateNode poolCI system

   Contributing factor

   Credential in configurationShared credential

   Detection

   A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

   Remediation

   • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
   • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again

   Action items

   • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
   • Rotate the password cleanly out of hours with a graceful recycle of the nodes

   Lessons

   The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

   View case

By root cause2