Cases3 cases

1 to 3 of 3

1. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
2. IL-0372 Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed

   Heroku2021

   Symptoms

   From 00:00 UTC on 25 February 2021, Heroku apps attached to about 1% of hobby-tier Postgres databases could not connect to their databases. Heroku first posted that it was investigating connectivity issues with Hobby tier databases at 03:39 UTC.

   Stale endpoint routingHigh error rate

   Trigger

   An upgrade process in an upstream system wrongly changed the host names of several shared database systems.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseService discovery

   Contributing factor

   –

   Remediation

   • Halted the upstream upgrade process
   • Updated the relevant systems with the new host names and published a new release on the affected apps
   • Gave customers a workaround of rotating database credentials with heroku pg:credentials:rotate

   Action items

   • Improve monitoring to detect similar incidents
   • Provide tooling and documentation that shorten recovery from a similar incident

   View case
3. IL-0351 Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hosts

   Heroku2017

   Symptoms

   Between 3 and 7 June 2017, Heroku customers in both the EU and US regions had periods of high latency: some requests failed and others took an excessive time to pass through the Heroku Routing Layer. All customers in the EU and US were affected at various times.

   High latencyHigh error rateLoad imbalance

   Root cause

   DNS or network fault

   Component

   Authoritative DNS servers and recordsCloud load balancer

   Contributing factor

   Unverified assumption

   Remediation

   • Found the sources of high-volume traffic and worked with those apps' owners to distribute their requests better
   • Replaced the legacy DNS records with new, corrected DNS records

   Action items

   • Make monitoring alert engineers sooner to unevenly distributed traffic
   • Investigate how to configure DNS and load balancers to reduce the effect of high-volume and bursting traffic
   • Work with caching Add-on partners on the DNS configuration issues that affect them, to achieve proper load balancing

   View case

By root cause2