Incident Library
Applied filters ComponentDatabaseSymptomHigh error rateOrganisationHerokuClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0372 Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed
    Heroku2021
    Symptoms

    From 00:00 UTC on 25 February 2021, Heroku apps attached to about 1% of hobby-tier Postgres databases could not connect to their databases. Heroku first posted that it was investigating connectivity issues with Hobby tier databases at 03:39 UTC.

    Stale endpoint routingHigh error rate
    Trigger

    An upgrade process in an upstream system wrongly changed the host names of several shared database systems.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseService discovery
    Contributing factor
    –
    Remediation
    • Halted the upstream upgrade process
    • Updated the relevant systems with the new host names and published a new release on the affected apps
    • Gave customers a workaround of rotating database credentials with heroku pg:credentials:rotate
    Action items
    • Improve monitoring to detect similar incidents
    • Provide tooling and documentation that shorten recovery from a similar incident
    View case
Check your cluster for this

This case involves Database and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
DatabaseService discovery
Contributing factor
–
Organisation
Heroku
  1. IL-0372 Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed
    Heroku2021