Cases2 cases

1 to 2 of 2

1. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes

   Railway2024

   Symptoms

   Railway's edge servers on Google Cloud could not route inbound requests to applications in any region for about 10 minutes, on 17 December 2024 UTC. Private networking, outbound requests, Metal Hosts and Dedicated Instances kept working.

   Service outage

   Trigger

   A merged Terraform change emptied the array of a static IP block list, to unblock a customer whose IP had been blocked since before Railway moved to a dynamic block list and who had reported intermittent timeouts.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unverified assumption

   Detection

   Automated alerting paged the current on-call engineer at 12:49 am UTC, two minutes after the change was merged.

   Remediation

   • Reverted the change to the IP allow-list rules
   • Applied the reverted change through the Terraform runner

   Action items

   • Automatically apply a page override for merged changes, for faster notification
   • Skip the Terraform plan phase during a revert
   • Remove the offending firewall rule safely
   • Audit all remaining legacy Google Cloud firewall rules for loose specificity
   • Move to its own metal infrastructure while keeping GCP and adding AWS, for resilience and fault isolation

   View case
2. IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

   cloud.gov2016

   Symptoms

   From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

   Network unreachableService outage

   Trigger

   At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

   Root cause

   Configuration change

   Component

   VPC route tables and firewall rules

   Contributing factor

   –

   Remediation

   • Added exceptions for individual applications, which restored some of them
   • Rolled back the security group change, which restored all applications

   Action items

   • Put all security group configuration into version control and make changes through pull requests
   • Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
   • Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
   • Create a notification system to reach the key contact of each application on the platform
   • Create a more accurate staging environment to test changes like this

   View case

By root cause1