IL-0354 Fly.io · 2026-09-23networking

Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes

A Fly.io machine's 6PN address has the form fdaa:netid:netid:hostid:hostid:machid:machid:2, with the host ID taken from the physical host where the machine was created. Since 6PN addresses became stable in 2025, a machine keeps that host ID wherever it runs, and when a worker is decommissioned its internal IP and host ID can be given to a new host of another kind. In 2026 Fly.io was moving 6PN routing from its custom eBPF program to standard Linux routing tables, routing machines by where they are rather than by their address. Gateway peers are the exception: interactive peers used by flyctl, for example for fly ssh console, have no identity in the routing state and can only be routed by the gateway's host ID, so Fly.io added a Linux route sending the gateway's whole /80 prefix per netid to the gateway host. When a separate project made the edges act as gateways for non-static peers, edges that had reused decommissioned workers' host IDs got /80 routes, which overrode the per-machine routing code because that code relies on machines not yet accessed locally having no route. Machines still carrying those host IDs lost 6PN.

Fly.io deleted the bad /80 routes from the routing tables on all affected hosts and rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes, within about 10 minutes. A follow-up fix added interactive peers only as /128 routes, but the real problem was the priority between the gateway logic and the worker logic, and the failure briefly recurred the next day. Fly.io then fixed it by always looking first for machines that carry a host's ID, whether or not the host is known to be a worker.

Severity: –
Scope: Private networking (6PN) for the subset of machines whose host IDs had been reused by edges
Users affected: –
Region: –
Time to detect: –
Time to resolve: 10m

Symptoms

On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

The affected machines were those whose 6PN address carries the host ID of a worker host that had since been decommissioned, where a newly provisioned edge had picked up that worker's internal IP address and so its host ID. When edges started to be treated as gateways, each edge received a route for its whole /80 gateway prefix, and traffic for those machines followed the edge's route, although the machines run on workers and never on an edge.

Fly.io's existing 6PN checks did not catch it, because all of them use machines that still sit on their original host.

Symptom Network unreachable Misrouted requests

Root cause

Root cause Faulty deployment

Component Host routes and routing rules Network control plane (SDN)

Contributing factor Unverified assumption Gap in automated tests

Trigger: Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

Detection

–

Remediation

Deleted the bad /80 routes from the routing table on all affected hosts
Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is

Action items

Give the machines used to test and alert on 6PN non-local 6PN addresses
Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers

Lessons

Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

Sources

https://fly.io/infra-log/gateways-broke-6pn/ · blog · archived copy

Related cases

IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026
same root cause · shared Network control plane (SDN)
IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tablesLinode · 2025
same root cause · shared Network control plane (SDN)
IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMsGoogle Cloud · 2023
same root cause · shared Network control plane (SDN)

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Host routes and routing rules, Network control plane (SDN) among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0354
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0354/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.