IL-0373 Heroku · 2024-07-27databaseobservability
Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days
A third-party vendor deployed a monitoring change with a memory leak to the monitoring agent on Heroku's Postgres database clusters. Available memory on 33 clusters in the EU (Dublin) region fell steadily until the databases became inaccessible, and some Essential-tier customers could not connect. The first outage, found through customer support tickets nine hours after it began, was fixed only temporarily on 28 July by allocating more resources and restarting the clusters. A second outage began on 30 July. At 16:47 UTC that day a support engineer from the vendor joined and confirmed that a monitoring agent using up all available resources caused both outages. The technology team provisioned the previous version of the vendor's software, which did not have the leak, memory was reclaimed and the clusters returned to a healthy state, and Heroku was declared out of impact at 17:56 UTC on 30 July; the resolving status update gives the second impact window as 06:04 to 16:28 UTC. The vendor provided a temporary fix and was working on a permanent one.
- Severity
- –
- Scope
- 33 Heroku Postgres clusters; some Essential-tier customers could not connect to their databases
- Users affected
- –
- Region
- EU (Dublin)
- Time to detect
- 9h 19m
- Time to resolve
- 15h 45m
Symptoms
From 22:31 UTC on 27 July 2024, 33 Heroku Postgres clusters in the EU (Dublin) region could not serve requests and ran into memory problems, and some Essential-tier customers could not connect to their databases. Customers trying to reach an affected database received out of memory errors.
Heroku Support paged the technology team with customer support tickets at 07:50 UTC on 28 July; at that point 33 databases were inaccessible or out of memory. Adding resources and restarting the clusters brought them back by 14:16 UTC on 28 July, but the problem returned on 30 July, when Heroku Support escalated it to the technology team at 06:04 UTC.
Symptom Memory pressureService outage
Root cause
Root cause Software defect
- Trigger
- A third-party vendor deployed a change to a monitoring agent that runs on Heroku's Postgres database clusters.
Detection
Heroku Support paged the technology team at 07:50 UTC on 28 July based on customer support tickets. The second occurrence also reached the technology team as an escalation from Heroku Support.
Remediation
- Allocated more resources and restarted the Postgres clusters, which fixed the first outage temporarily
- Provisioned the previous version of the third-party monitoring software, which did not leak memory
- The vendor applied an interim mitigation
Action items
- Add enhanced metrics collection for the third-party monitoring software
- Evaluate the vendor's fixed release of the monitoring software and roll it out if acceptable
Lessons
–
Sources
- https://status.heroku.com/incidents/2678 · official-status · archived copy
Related cases
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0247One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to KubernetesPinterest · 2025same root cause · shared Telemetry agent
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared Database
IL-0174After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its typeHyperconnect · 2023same root cause · shared Telemetry agent
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0041fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020same root cause · shared Telemetry agent
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0051A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open filesthredUP · 2018same root cause · shared Telemetry agent
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0373
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0373/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.