IL-0018 UK Ministry of Justice · 2023-01-11kubernetesawssecurity
On 11 January 2023 the Docker Hub password used by the Ministry of Justice Cloud Platform was changed as part of LastPass remediation activities. Concourse and the EKS cluster nodes kept logging in with the old password, Docker Hub refused them for too many failed login attempts, and neither could pull images from Docker Hub for 1h 28m.
The team updated the new password in the cloud-platform-infrastructure repository for Concourse at 14:40 and for the EKS cluster at 14:56. At 15:01 the Concourse plan showed that the change would update the launch template, which suggested a node recycle. The team applied the update to the live-2 cluster to find out whether a recycle would be needed, and at 15:28 confirmed that the update would force node recycles across the live and manager clusters. At 15:36 it decided to put the previous Docker Hub password back so that a clean rotation could be done out of hours. The password was changed back at 15:40, the reporting user confirmed at 15:46 that the pod was deploying, and image pulls were working again at 15:50. live-2 was left with an invalid dockerconfig, which had no impact on users.
As the review action, the platform switched from a Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform.
At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address
At 14:34 the team compared the cluster's Docker Hub passwords with the value stored in LastPass, and at 14:40 updated the Concourse Docker Hub password in the cloud-platform-infrastructure terraform.tfvars. At 14:51 the explanation came out: the Docker Hub password had been changed as part of LastPass remediation activities.
The problem then showed on the clusters. At 14:52 the Kuberhealthy daemonset check reported that the cluster could not pull images either, with the error "Check execution error: kuberhealthy/daemonset: error when waiting for pod to start: ErrImagePull". At 14:54 a user reported ErrImagePull when creating port-forward pods, in at least two namespaces.
Updating the password for the nodes was not a quick fix. The node dockerconfig had to change, and at 15:01 the Concourse plan showed that the password update would change the launch template. At 15:28 the team confirmed that the update would force node recycles across the live and manager clusters.
Symptom Image pull failures
A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.
The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.
IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.