Cases5 cases

1 to 5 of 5

1. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes

   Railway2024

   Symptoms

   Railway's edge servers on Google Cloud could not route inbound requests to applications in any region for about 10 minutes, on 17 December 2024 UTC. Private networking, outbound requests, Metal Hosts and Dedicated Instances kept working.

   Service outage

   Trigger

   A merged Terraform change emptied the array of a static IP block list, to unblock a customer whose IP had been blocked since before Railway moved to a dynamic block list and who had reported intermittent timeouts.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unverified assumption

   Detection

   Automated alerting paged the current on-call engineer at 12:49 am UTC, two minutes after the change was merged.

   Remediation

   • Reverted the change to the IP allow-list rules
   • Applied the reverted change through the Terraform runner

   Action items

   • Automatically apply a page override for merged changes, for faster notification
   • Skip the Terraform plan phase during a revert
   • Remove the offending firewall rule safely
   • Audit all remaining legacy Google Cloud firewall rules for loose specificity
   • Move to its own metal infrastructure while keeping GCP and adding AWS, for resilience and fault isolation

   View case
2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

   UK Ministry of Justice2024

   Symptoms

   On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

   Service outageNetwork unreachableResources deleted

   Trigger

   At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

   Root cause

   Operator error

   Component

   TerraformCI systemVPC route tables and firewall rules

   Contributing factor

   No safeguard on destructive change

   Detection

   High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

   Remediation

   • Ran terraform apply locally to rebuild the subnet route table associations

   Action items

   • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
   • Find a way to hold queued pull requests while manual infrastructure updates are under way
   • Find ways to isolate the plan and apply tasks of infrastructure pipelines better

   View case
3. IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them

   Twingate2023

   Symptoms

   At 01:40 UTC on 14 February 2023, Twingate saw the number of connections drop in two of its Relay clusters, the one in us-east4 (Ashburn, Virginia) and the one in europe-west6 (Zurich, Switzerland).

   Network unreachable

   Trigger

   At 01:00 UTC on 14 February, Twingate started a Terraform change to all of its Relay clusters worldwide to add spot instance scaling; the rollout took about 45 minutes.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unrepresentative stagingNo staged rollout

   Detection

   At 01:40 UTC Twingate noticed a decrease in the number of connections in the two clusters and started investigating.

   Remediation

   • Disabled the pods in the two affected Relay clusters at 02:04 UTC so Clients and Connectors reconnected to the next closest Relay cluster
   • Corrected the Terraform configuration error and redeployed the affected clusters at 03:30 UTC

   Action items

   • Make the staging environment reflect the sizing and configuration differences between Relay clusters
   • Create Terraform plans automatically for every environment and configuration variation on feature development branches
   • Extend Relay health checks so a cluster is not discoverable to Connectors and Clients when its network firewall tag is missing
   • Research and introduce staggered Terraform rollouts for the Relay cluster infrastructure
   • Add multi-cluster connectivity to Connectors so they handle regional Relay cluster problems automatically

   View case
4. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers

   Acquia2019

   Symptoms

   On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

   Service outageNetwork unreachable

   Trigger

   While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

   Root cause

   Operator error

   Component

   VPC route tables and firewall rulesIn-memory cache

   Contributing factor

   Ignored early signal

   Detection

   Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

   Remediation

   • Restored the missing rule in the security group by hand

   Action items

   • Correct the internal workflow with all involved teams to prevent similar mistakes
   • Alert internal teams faster about configuration drift in VPC security groups

   View case
5. IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

   cloud.gov2016

   Symptoms

   From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

   Network unreachableService outage

   Trigger

   At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

   Root cause

   Configuration change

   Component

   VPC route tables and firewall rules

   Contributing factor

   –

   Remediation

   • Added exceptions for individual applications, which restored some of them
   • Rolled back the security group change, which restored all applications

   Action items

   • Put all security group configuration into version control and make changes through pull requests
   • Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
   • Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
   • Create a notification system to reach the key contact of each application on the platform
   • Create a more accurate staging environment to test changes like this

   View case

By root cause2