Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
    Symptoms

    On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

    Service outageNetwork unreachableResources deleted
    Trigger

    At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

    Root cause
    Operator error
    Component
    TerraformCI systemVPC route tables and firewall rules
    Contributing factor
    No safeguard on destructive change
    Detection

    High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

    Remediation
    • Ran terraform apply locally to rebuild the subnet route table associations
    Action items
    • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
    • Find a way to hold queued pull requests while manual infrastructure updates are under way
    • Find ways to isolate the plan and apply tasks of infrastructure pipelines better
    View case
  2. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
    Acquia2019
    Symptoms

    On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

    Service outageNetwork unreachable
    Trigger

    While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

    Root cause
    Operator error
    Component
    VPC route tables and firewall rulesIn-memory cache
    Contributing factor
    Ignored early signal
    Detection

    Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

    Remediation
    • Restored the missing rule in the security group by hand
    Action items
    • Correct the internal workflow with all involved teams to prevent similar mistakes
    • Alert internal teams faster about configuration drift in VPC security groups
    View case
Check your cluster for this

These 2 cases involve VPC route tables and firewall rules and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error2 cases

Component
VPC route tables and firewall rules2In-memory cache1CI system1Terraform1
Contributing factor
Ignored early signal1No safeguard on destructive change1
Organisation
Acquia1UK Ministry of Justice1
  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  2. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
    Acquia2019