Incident Library
Applied filters SymptomMisrouted requestsFactorGap in automated testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes
    Fly.io2026
    Symptoms

    On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

    Network unreachableMisrouted requests
    Trigger

    Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

    Root cause
    Faulty deployment
    Component
    Host routes and routing rulesNetwork control plane (SDN)
    Contributing factor
    Unverified assumptionGap in automated tests
    Remediation
    • Deleted the bad /80 routes from the routing table on all affected hosts
    • Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
    • Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
    • Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is
    Action items
    • Give the machines used to test and alert on 6PN non-local 6PN addresses
    • Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
    • Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers
    Lessons

    Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

    View case
Check your cluster for this

This case involves Host routes and routing rules and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Host routes and routing rulesNetwork control plane (SDN)
Contributing factor
Unverified assumptionGap in automated tests
Organisation
Fly.io
  1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes
    Fly.io2026