Incident Library

IL-0352 Heroku · 2025-07-18networking

Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts

Router 2.0, the default router of Heroku's Common Runtime, uses entropy metadata to spread dyno selection. A code condition present since the Router 2.0 beta reduced the variance of that selection, and a router instance that hit it kept balancing badly for all later requests. Hitting it became more likely as request volume on Router 2.0 grew with the migration of about 94% of Common Runtime apps. From 22:00 UTC on 18 July 2025 more and more router instances hit the condition, and requests piled onto some dynos of affected apps, which caused latency and timeouts. The bug also showed in staging load tests, but their alerts watch availability and latency, not request distribution, so nothing fired, and Heroku learned of the problem only from customers on 20 July.

On 22 July Heroku declared an internal investigation at 14:35 UTC and a formal incident at 17:48 UTC, posted to the status site at 18:36 UTC, and advised customers to fall back to the legacy router. After seeing the same pattern in staging, engineers cycled the production router fleets at 19:42 UTC, which improved balancing, identified the broken entropy metadata at 20:05 UTC, and at 21:01 UTC reproduced the fault with a new load balancing benchmark test and prepared a fix that passed it. The fix was deployed to staging at 21:13 UTC, judged safe at 21:30 UTC, rolled out to production from 21:38 UTC in stages, and reached the last partition at 00:12 UTC on 23 July, which ended the incident.

Severity
–
Scope
HTTP request distribution across web dynos for apps using Router 2.0 on the Common Runtime, with higher latency, timeouts, reduced availability and Heroku error codes
Users affected
Some customer applications using Router 2.0 on Heroku's Common Runtime
Region
–
Time to detect
42h 46m
Time to resolve
98h 12m

Symptoms

From about 22:00 UTC on 18 July 2025, some apps on Heroku's Common Runtime that used Router 2.0 had their HTTP requests balanced unevenly across their web dynos. The dynos that got more requests were overloaded, and the apps saw higher HTTP request latency, timeouts, reduced availability and various Heroku error codes in application logs and dashboards.

The imbalance, measured as the percentage variance in HTTP requests per dyno, grew steadily over the following days. Once a router instance hit the faulty code condition, its load balancing stayed broken for every later request, and more router instances hit the condition over time.

Heroku Support received the first customer reports of uneven request distribution, with higher latency and timeouts, at 16:46 UTC on 20 July. An internal investigation was declared at 14:35 UTC on 22 July, and Support began advising affected customers to disable Router 2.0 and fall back to the legacy router. Engineers first suspected incomplete router dyno state and dropped that theory at 17:01 UTC. At 17:53 UTC they saw the same uneven distribution in the logs of the router's continuous load tests in staging, turned to the load balancing metadata, and found at 19:42 UTC that cycling router instances improved balancing.

The staging load tests had not raised an alert. They alert and block deployments on reduced routing availability and higher latency, but request distribution was not an alerting signal, and the uneven distribution in staging did not strain the test apps enough to move availability or latency.

Symptom Load imbalanceHigh latencyIntermittent timeouts

Root cause

Root cause Software defect

Component Cloud load balancer
Contributing factor Monitoring blind spotUnrepresentative staging
Trigger
From 22:00 UTC on 18 July 2025 router instances began hitting a code condition that broke the entropy metadata Router 2.0 uses to choose dynos. The condition had been in the code since the Router 2.0 beta and became more likely as more HTTP requests went through the new router, with about 94% of Common Runtime apps migrated.

Detection

No alert fired. Heroku Support received the first customer reports of uneven request balancing at 16:46 UTC on 20 July 2025, and an internal investigation was declared on 22 July.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.