IL-0519 cloud.gov · 2016-03-08awsnetworking

Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

On 8 March 2016 at 9:28am Pacific Time, a contractor at Aquilent changed the AWS security group settings of the cloud.gov PaaS as part of a set of changes toward ATO for the platform. The change restricted outbound traffic, but several applications on cloud.gov needed outbound connections to reach services they depended on, so they became unavailable, and at one point all applications were down. The team added exceptions for individual applications, which restored some of them, and then decided to roll back the change, which restored all applications at 8:00am Pacific Time on 9 March. A postmortem was held on 10 March. In the issue comments a contributor noted that moving from AWS security groups to CF security groups would be a large change, because AWS security groups are a core part of the platform's FedRAMP Authorization. The issue was closed in June 2016 without the planned review, because procedures and the contractor had changed since.

Severity: –
Scope: Applications on cloud.gov that relied on outbound traffic; at one point all applications on the platform
Users affected: –
Region: –
Time to detect: –
Time to resolve: 22h 32m

Symptoms

From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

Part of the outbound traffic from the platform was blocked by a change to the AWS security group settings. Adding exceptions for individual applications brought some of them back, but not all.

Symptom Network unreachable Service outage

Root cause

Root cause Configuration change

Component VPC route tables and firewall rules

Contributing factor –

Trigger: At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

Detection

–

Remediation

Added exceptions for individual applications, which restored some of them
Rolled back the security group change, which restored all applications

Action items

Put all security group configuration into version control and make changes through pull requests
Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
Create a notification system to reach the key contact of each application on the platform
Create a more accurate staging environment to test changes like this

Lessons

–

Sources

https://github.com/cloud-gov/cg-postmortems/issues/2 · github · archived copy

Related cases

IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutesRailway · 2024
same root cause · shared VPC route tables and firewall rules
IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to themTwingate · 2023
same root cause · shared VPC route tables and firewall rules

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, VPC route tables and firewall rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0519
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0519/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.