Cases2 cases

1 to 2 of 2

1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes

   Fly.io2026

   Symptoms

   On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

   Network unreachableMisrouted requests

   Trigger

   Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

   Root cause

   Faulty deployment

   Component

   Host routes and routing rulesNetwork control plane (SDN)

   Contributing factor

   Unverified assumptionGap in automated tests

   Remediation

   • Deleted the bad /80 routes from the routing table on all affected hosts
   • Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
   • Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
   • Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is

   Action items

   • Give the machines used to test and alert on 6PN non-local 6PN addresses
   • Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
   • Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers

   Lessons

   Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

   View case
2. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case

By root cause2