IL-0227 Heroku · 2025-06-10networking

Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

Beginning at 06:00 UTC on 10 June 2025, Heroku customers had a service disruption that meant up to 24 hours of downtime for many of them; it was not a security incident, and no customer data was lost. The networking service on Heroku's hosts relied on a legacy script that applied the correct routing rules only on initial boot. When the automated update restarted the service, the routes were not applied again, and every dyno on the host lost outbound network connectivity. Heroku's internal tools and the Heroku Status Page ran on the same affected infrastructure, so the outage also hampered Heroku's response and its communication with customers. Heroku names a lack of immutability controls, which let an automated process change production, and gaps in tooling and process, which slowed diagnosis.

From 12:56 UTC the team restarted internal instances in rotation, which gave some relief, and from 13:58 UTC it posted updates to the @herokustatus account on X. Heroku asked an upstream vendor to invalidate the token used for the automated updates; this was confirmed at 17:30 UTC and completed at 19:18 UTC, so no further hosts were hit, and the automated upgrade service was disabled. The Heroku Dashboard was fully restored by 20:59 UTC, a fleetwide dyno recycle started at 22:01 UTC, and after downstream systems caught up the incident was declared resolved at 05:50 UTC on 11 June.

Severity: –
Scope: Heroku applications: most dynos in Private Spaces without outbound connectivity, intermittent logins, application failures, the Heroku Dashboard and the Heroku Status Page
Users affected: many customers, with up to 24 hours of downtime
Region: –
Time to detect: –
Time to resolve: 23h 50m

Symptoms

From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

By 08:26 UTC the investigation confirmed the core problem: most dynos in Private Spaces could not make outbound HTTP requests. The team ruled out an upstream provider issue and placed the failure in Heroku's own infrastructure, and comparing healthy and unhealthy hosts showed missing network routes at 11:54 UTC.

Only at 13:11 UTC did the team learn of an unexpected restart of the network service, and at 13:42 UTC it found the trigger, an automated upgrade of a system package. Many customers had up to 24 hours of downtime. After service came back, delayed work kept arriving for hours: status emails from early in the incident were delivered late, Heroku Connect syncing had to catch up, and the Heroku release phase had a long backlog that took a few hours to clear. The incident was declared resolved at 05:50 UTC on 11 June.

Symptom Network unreachable Service outage High error rate

Root cause

Root cause Breaking change in an upgrade

Component Host routes and routing rules

Contributing factor Unpinned version Circular dependency Limited debug access

Trigger: An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

Detection

Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

Remediation

Restarted internal instances in rotation, which gave some relief
Had an upstream vendor invalidate the token used for the automated updates
Disabled the automated upgrade service
Recycled dynos across the whole fleet

Action items

Put permanent controls in place so no system change happens outside the controlled deployment process
Audit all base images for similar risks
Make network routing survive graceful restarts of the networking service
Build fully independent backup communication channels
Overhaul incident response tooling, including tools that run queries across the whole fleet
Streamline break-glass procedures for rapid access to critical systems
Improve monitoring to detect complex issues earlier

Lessons

Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

Sources

https://www.heroku.com/blog/summary-of-june-10-outage/ · blog · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Host routes and routing rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0227
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0227/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.