Incident Library

Applied filters SymptomNetwork unreachableOrganisationAcquiaClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases1 case

1 to 1 of 1

IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
Acquia2019
Symptoms
On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.
Service outage Network unreachable
Trigger
While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.
Root cause
Operator error
Component
VPC route tables and firewall rules In-memory cache
Contributing factor
Ignored early signal
Detection
Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.
Remediation
Restored the missing rule in the security group by hand
Action items
Correct the internal workflow with all involved teams to prevent similar mistakes
Alert internal teams faster about configuration drift in VPC security groups
View case

Check your cluster for this

This case involves In-memory cache and VPC route tables and firewall rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component: In-memory cache VPC route tables and firewall rules
Contributing factor: Ignored early signal
Organisation: Acquia

IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
Acquia2019