Cases2 cases

1 to 2 of 2

1. IL-0373 Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days

   Heroku2024

   Symptoms

   From 22:31 UTC on 27 July 2024, 33 Heroku Postgres clusters in the EU (Dublin) region could not serve requests and ran into memory problems, and some Essential-tier customers could not connect to their databases. Customers trying to reach an affected database received out of memory errors.

   Memory pressureService outage

   Trigger

   A third-party vendor deployed a change to a monitoring agent that runs on Heroku's Postgres database clusters.

   Root cause

   Software defect

   Component

   DatabaseTelemetry agent

   Contributing factor

   Monitoring blind spot

   Detection

   Heroku Support paged the technology team at 07:50 UTC on 28 July based on customer support tickets. The second occurrence also reached the technology team as an escalation from Heroku Support.

   Remediation

   • Allocated more resources and restarted the Postgres clusters, which fixed the first outage temporarily
   • Provisioned the previous version of the third-party monitoring software, which did not leak memory
   • The vendor applied an interim mitigation

   Action items

   • Add enhanced metrics collection for the third-party monitoring software
   • Evaluate the vendor's fixed release of the monitoring software and roll it out if acceptable

   View case
2. IL-0372 Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed

   Heroku2021

   Symptoms

   From 00:00 UTC on 25 February 2021, Heroku apps attached to about 1% of hobby-tier Postgres databases could not connect to their databases. Heroku first posted that it was investigating connectivity issues with Hobby tier databases at 03:39 UTC.

   Stale endpoint routingHigh error rate

   Trigger

   An upgrade process in an upstream system wrongly changed the host names of several shared database systems.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseService discovery

   Contributing factor

   –

   Remediation

   • Halted the upstream upgrade process
   • Updated the relevant systems with the new host names and published a new release on the affected apps
   • Gave customers a workaround of rotating database credentials with heroku pg:credentials:rotate

   Action items

   • Improve monitoring to detect similar incidents
   • Provide tooling and documentation that shorten recovery from a similar incident

   View case

By root cause2