Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed

Symptoms

From 00:00 UTC on 25 February 2021, Heroku apps attached to about 1% of hobby-tier Postgres databases could not connect to their databases. Heroku first posted that it was investigating connectivity issues with Hobby tier databases at 03:39 UTC.

The connection strings of the affected databases had changed, but the attached apps still had the old ones. While the fix was rolling out, a database owner could bring the database back by rotating its credentials with `heroku pg:credentials:rotate`, adding the `--all` flag when the database had several credentials.

Symptom Stale endpoint routingHigh error rate

Root cause

Trigger

An upgrade process in an upstream system wrongly changed the host names of several shared database systems.

Detection

–

Remediation

• Halted the upstream upgrade process
• Updated the relevant systems with the new host names and published a new release on the affected apps
• Gave customers a workaround of rotating database credentials with heroku pg:credentials:rotate

Action items

• Improve monitoring to detect similar incidents
• Provide tooling and documentation that shorten recovery from a similar incident

Lessons

–

Sources

• https://status.heroku.com/incidents/2208 · official-status · archived copy

Related cases

• IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a dayMercari · 2026

  same root cause · shared Database

• IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026

  same root cause · shared Database

• IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shardsIntercom · 2026

  same root cause · shared Database

• IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memoryRailway · 2025

  same root cause · shared Database

• IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four daysClerk · 2025

  same root cause · shared Database

• IL-0444 Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ranRepro · 2024

  same root cause · shared Database

• IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

  same root cause · shared Database

• IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restartingDaily · 2023

  same root cause · shared Database

• IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slowRevenueCat · 2022

  same root cause · shared Database

• IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI · 2021

  same root cause · shared Database

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0372
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0372/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.