Cases3 cases

1 to 3 of 3

1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes

   Fly.io2026

   Symptoms

   On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

   Network unreachableMisrouted requests

   Trigger

   Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

   Root cause

   Faulty deployment

   Component

   Host routes and routing rulesNetwork control plane (SDN)

   Contributing factor

   Unverified assumptionGap in automated tests

   Remediation

   • Deleted the bad /80 routes from the routing table on all affected hosts
   • Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
   • Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
   • Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is

   Action items

   • Give the machines used to test and alert on 6PN non-local 6PN addresses
   • Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
   • Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers

   Lessons

   Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

   View case
2. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs

   Infrawaves2026

   Symptoms

   In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

   Link flappingTraining job failure

   Trigger

   Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

   Root cause

   Software defect

   Component

   NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules

   Contributing factor

   Unreviewed defaults

   Remediation

   • Configured the IP addresses with the Ubuntu Networking service instead of NetworkManager, so a port down no longer removed the address or changed the GID

   Lessons

   Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

   View case
3. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case

By root cause3