Cases2 cases

1 to 2 of 2

1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

   UK Ministry of Justice2023

   Symptoms

   At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

   Image pull failures

   Trigger

   The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

   Root cause

   Operator error

   Component

   Container registry and its credentialsNode launch templateNode poolCI system

   Contributing factor

   Credential in configurationShared credential

   Detection

   A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

   Remediation

   • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
   • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again

   Action items

   • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
   • Rotate the password cleanly out of hours with a graceful recycle of the nodes

   Lessons

   The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

   View case
2. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case

By root cause1