Incident Library
Applied filters Root causeOperator errorComponentNode poolFactorShared credentialClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
    Symptoms

    At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

    Image pull failures
    Trigger

    The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

    Root cause
    Operator error
    Component
    Container registry and its credentialsNode launch templateNode poolCI system
    Contributing factor
    Credential in configurationShared credential
    Detection

    A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

    Remediation
    • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
    • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again
    Action items
    • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
    • Rotate the password cleanly out of hours with a graceful recycle of the nodes
    Lessons

    The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

    View case
Check your cluster for this

This case involves CI system and Node launch template. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
CI systemNode launch templateNode poolContainer registry and its credentials
Contributing factor
Credential in configurationShared credential
Organisation
UK Ministry of Justice
  1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023