Incident Library
Applied filters SymptomService outageFactorUnverified assumptionOrganisationRailwayClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes
    Railway2024
    Symptoms

    Railway's edge servers on Google Cloud could not route inbound requests to applications in any region for about 10 minutes, on 17 December 2024 UTC. Private networking, outbound requests, Metal Hosts and Dedicated Instances kept working.

    Service outage
    Trigger

    A merged Terraform change emptied the array of a static IP block list, to unblock a customer whose IP had been blocked since before Railway moved to a dynamic block list and who had reported intermittent timeouts.

    Root cause
    Configuration change
    Component
    TerraformVPC route tables and firewall rules
    Contributing factor
    Unverified assumption
    Detection

    Automated alerting paged the current on-call engineer at 12:49 am UTC, two minutes after the change was merged.

    Remediation
    • Reverted the change to the IP allow-list rules
    • Applied the reverted change through the Terraform runner
    Action items
    • Automatically apply a page override for merged changes, for faster notification
    • Skip the Terraform plan phase during a revert
    • Remove the offending firewall rule safely
    • Audit all remaining legacy Google Cloud firewall rules for loose specificity
    • Move to its own metal infrastructure while keeping GCP and adding AWS, for resilience and fault isolation
    View case
Check your cluster for this

This case involves Terraform and VPC route tables and firewall rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
TerraformVPC route tables and firewall rules
Contributing factor
Unverified assumption
Organisation
Railway
  1. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes
    Railway2024