Incident Library
Applied filters ComponentTerraformSymptomNetwork unreachableDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
    Symptoms

    On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

    Service outageNetwork unreachableResources deleted
    Trigger

    At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

    Root cause
    Operator error
    Component
    TerraformCI systemVPC route tables and firewall rules
    Contributing factor
    No safeguard on destructive change
    Detection

    High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

    Remediation
    • Ran terraform apply locally to rebuild the subnet route table associations
    Action items
    • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
    • Find a way to hold queued pull requests while manual infrastructure updates are under way
    • Find ways to isolate the plan and apply tasks of infrastructure pipelines better
    View case
Check your cluster for this

This case involves CI system and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
CI systemTerraformVPC route tables and firewall rules
Contributing factor
No safeguard on destructive change
Organisation
UK Ministry of Justice
  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024