Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
    Symptoms

    On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

    Service outageNetwork unreachableResources deleted
    Trigger

    At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

    Root cause
    Operator error
    Component
    TerraformCI systemVPC route tables and firewall rules
    Contributing factor
    No safeguard on destructive change
    Detection

    High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

    Remediation
    • Ran terraform apply locally to rebuild the subnet route table associations
    Action items
    • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
    • Find a way to hold queued pull requests while manual infrastructure updates are under way
    • Find ways to isolate the plan and apply tasks of infrastructure pipelines better
    View case
  2. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
    Symptoms

    At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

    Image pull failures
    Trigger

    The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

    Root cause
    Operator error
    Component
    Container registry and its credentialsNode launch templateNode poolCI system
    Contributing factor
    Credential in configurationShared credential
    Detection

    A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

    Remediation
    • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
    • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again
    Action items
    • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
    • Rotate the password cleanly out of hours with a graceful recycle of the nodes
    Lessons

    The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

    View case
  3. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
Check your cluster for this

These 3 cases involve CI system and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error3 cases

Component
CI system2Cloud load balancer1Credential broker for pods1external-dns1Ingress controller1Node launch template1Node pool1Prometheus1Container registry and its credentials1Terraform1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change2Credential in configuration1Shared credential1
Organisation
UK Ministry of Justice3
  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  2. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
  3. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020