Cases2 cases

1 to 2 of 2

1. IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts

   Heroku2025

   Symptoms

   From about 22:00 UTC on 18 July 2025, some apps on Heroku's Common Runtime that used Router 2.0 had their HTTP requests balanced unevenly across their web dynos. The dynos that got more requests were overloaded, and the apps saw higher HTTP request latency, timeouts, reduced availability and various Heroku error codes in application logs and dashboards.

   Load imbalanceHigh latencyIntermittent timeouts

   Trigger

   From 22:00 UTC on 18 July 2025 router instances began hitting a code condition that broke the entropy metadata Router 2.0 uses to choose dynos. The condition had been in the code since the Router 2.0 beta and became more likely as more HTTP requests went through the new router, with about 94% of Common Runtime apps migrated.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative staging

   Detection

   No alert fired. Heroku Support received the first customer reports of uneven request balancing at 16:46 UTC on 20 July 2025, and an internal investigation was declared on 22 July.

   Remediation

   • Advised affected customers to disable Router 2.0 and fall back to the legacy router
   • Cycled the production router fleets, which improved load balancing for affected apps
   • Deployed the router fix to staging, then rolled it out to all Common Runtime partitions by 00:12 UTC on 23 July

   Action items

   • Introduced gating and notifications so that load balancing behaviour regressions do not occur
   • Communicate the timeline and send reminders to apps still on the legacy router before automated migrations resume

   View case
2. IL-0373 Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days

   Heroku2024

   Symptoms

   From 22:31 UTC on 27 July 2024, 33 Heroku Postgres clusters in the EU (Dublin) region could not serve requests and ran into memory problems, and some Essential-tier customers could not connect to their databases. Customers trying to reach an affected database received out of memory errors.

   Memory pressureService outage

   Trigger

   A third-party vendor deployed a change to a monitoring agent that runs on Heroku's Postgres database clusters.

   Root cause

   Software defect

   Component

   DatabaseTelemetry agent

   Contributing factor

   Monitoring blind spot

   Detection

   Heroku Support paged the technology team at 07:50 UTC on 28 July based on customer support tickets. The second occurrence also reached the technology team as an escalation from Heroku Support.

   Remediation

   • Allocated more resources and restarted the Postgres clusters, which fixed the first outage temporarily
   • Provisioned the previous version of the third-party monitoring software, which did not leak memory
   • The vendor applied an interim mitigation

   Action items

   • Add enhanced metrics collection for the third-party monitoring software
   • Evaluate the vendor's fixed release of the monitoring software and roll it out if acceptable

   View case

By root cause1