Incident Library
Applied filters Root causeOperator errorComponentIn-memory cacheFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
    Acquia2019
    Symptoms

    On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

    Service outageNetwork unreachable
    Trigger

    While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

    Root cause
    Operator error
    Component
    VPC route tables and firewall rulesIn-memory cache
    Contributing factor
    Ignored early signal
    Detection

    Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

    Remediation
    • Restored the missing rule in the security group by hand
    Action items
    • Correct the internal workflow with all involved teams to prevent similar mistakes
    • Alert internal teams faster about configuration drift in VPC security groups
    View case
Check your cluster for this

This case involves In-memory cache and VPC route tables and firewall rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
In-memory cacheVPC route tables and firewall rules
Contributing factor
Ignored early signal
Organisation
Acquia
  1. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
    Acquia2019