IL-0245 Acquia · 2019-02-07networking

Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers

Acquia was putting in a temporary change to capture system diagnostics that are not part of its standard package, which meant setting up a temporary file system for the diagnostic data. During that work a rule was removed by mistake from a security group in a shared VPC in US-East. The rule allowed web servers to connect to the Memcache servers; without it PHP processes on the web servers were blocked and sites became unavailable. Acquia's configuration drift alerting worked, but the process around drift errors meant the alert was not seen in time to shorten the outage. Acquia restored the missing security group rule by hand, and the outage lasted from 22:17 to 23:25 UTC.

Severity: –
Scope: Acquia Cloud Enterprise sites (Cloud Platform Enterprise) in US-East
Users affected: multiple customers
Region: US-East
Time to detect: –
Time to resolve: 1h 8m

Symptoms

On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

Symptom Service outage Network unreachable

Root cause

Root cause Operator error

Component VPC route tables and firewall rules In-memory cache

Contributing factor Ignored early signal

Trigger: While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

Detection

Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

Remediation

Restored the missing rule in the security group by hand

Action items

Correct the internal workflow with all involved teams to prevent similar mistakes
Alert internal teams faster about configuration drift in VPC security groups

Lessons

–

Sources

https://status.acquia.com/incidents/15g5548q4s1h · official-status

Related cases

IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offlineUK Ministry of Justice · 2024
same root cause · shared VPC route tables and firewall rules

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, VPC route tables and firewall rules, In-memory cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0245
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0245/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.