IL-0355 Railway · 2024-12-17networkinggcp

Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes

Railway had kept both a static and a dynamic IP block list and over two years moved fully to the dynamic one. A customer reported intermittent timeouts when talking to Railway, and its IP turned out to be on an old static list. Railway removed it by emptying the list's array in Terraform. The Google Cloud Terraform provider treats source_ranges as an optional list, and Terraform, written in Go, does not tell an empty array from nil, so the change removed the source IP list from the GCP firewall rule. A GCP firewall rule without a source range selects all IPs, so the rule meant to block an empty list of IPs blocked everything, and all inbound traffic to Railway's Google Cloud edge was cut. The page reached the current on-call engineer, waking them, rather than the engineer who had merged the change, and the bastion Terraform runner ran a plan before applying the revert, adding about 3 minutes. The revert was applied at 12:58 am UTC and monitoring recovered at 12:59 am.

Severity: –
Scope: Inbound requests through Railway's Google Cloud edge servers to applications; private networking, outbound requests, Metal Hosts and Dedicated Instances were not affected
Users affected: –
Region: All Railway regions on Google Cloud
Time to detect: 2m
Time to resolve: 12m

Symptoms

Railway's edge servers on Google Cloud could not route inbound requests to applications in any region for about 10 minutes, on 17 December 2024 UTC. Private networking, outbound requests, Metal Hosts and Dedicated Instances kept working.

Automated alerting notified a Railway engineer at 12:49 am UTC, two minutes after a change to IP allow-list rules was merged at 12:47 am, and automated monitoring recovered at 12:59 am after the change was reverted and applied.

Symptom Service outage

Root cause

Root cause Configuration change

Component Terraform VPC route tables and firewall rules

Contributing factor Unverified assumption

Trigger: A merged Terraform change emptied the array of a static IP block list, to unblock a customer whose IP had been blocked since before Railway moved to a dynamic block list and who had reported intermittent timeouts.

Detection

Automated alerting paged the current on-call engineer at 12:49 am UTC, two minutes after the change was merged.

Remediation

Reverted the change to the IP allow-list rules
Applied the reverted change through the Terraform runner

Action items

Automatically apply a page override for merged changes, for faster notification
Skip the Terraform plan phase during a revert
Remove the offending firewall rule safely
Audit all remaining legacy Google Cloud firewall rules for loose specificity
Move to its own metal infrastructure while keeping GCP and adding AWS, for resilience and fault isolation

Lessons

–

Sources

https://blog.railway.com/p/incident-december-16-2024 · blog · archived copy

Related cases

IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to themTwingate · 2023
same root cause · shared Terraform, VPC route tables and firewall rules
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023
same root cause · shared Terraform
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
same root cause · shared Terraform
IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023
same root cause · shared Terraform
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021
same root cause · shared Terraform
IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020
same root cause · shared Terraform
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019
same root cause · shared Terraform
IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound trafficcloud.gov · 2016
same root cause · shared VPC route tables and firewall rules

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Terraform, VPC route tables and firewall rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0355
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0355/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.