Cases1 case

1 to 1 of 1

1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes

   Fly.io2026

   Symptoms

   On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

   Network unreachableMisrouted requests

   Trigger

   Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

   Root cause

   Faulty deployment

   Component

   Host routes and routing rulesNetwork control plane (SDN)

   Contributing factor

   Unverified assumptionGap in automated tests

   Remediation

   • Deleted the bad /80 routes from the routing table on all affected hosts
   • Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
   • Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
   • Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is

   Action items

   • Give the machines used to test and alert on 6PN non-local 6PN addresses
   • Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
   • Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers

   Lessons

   Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

   View case

By root cause1