OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc error

Symptoms

At 21:40 UTC on 25 June 2025, Alertmanager alerted the OpenStreetMap operations team: minutely replication feed delayed - 6m 2s. Since 21:33 UTC every run of the osmdbt timer that produces the minutely replication diffs had failed with ERROR: invalid memory alloc request size 1243650064.

The error came from PostgreSQL, in the query that osmdbt-get-log runs on the logical replication slot "osmdbt": SELECT * FROM pg_logical_slot_peek_changes($1, NULL, NULL). No minutely, hourly or daily diffs were produced, so tile.openstreetmap.org, Nominatim, Overpass and third-party services stopped receiving map edits. None of the alerts was a pager or SMS alert; the first community report came at 00:37 UTC on 26 June and the first social media report at 01:13 UTC.

At 06:08 UTC on 26 June the operations team stopped the osmdbt timer and ran osmdbt by hand with a reduced maximum number of changesets. Five diffs succeeded, then the error returned, until even --max-changes=1 triggered it. The team and the osmdbt developer looked at whether a huge set of map changes, such as 15 million or more node changes, had overwhelmed the PostgreSQL internal replication logs that osmdbt reads.

Symptom Replication lag

Root cause

Trigger

–

Detection

Alertmanager sent the first alert at 21:40 UTC, minutely replication feed delayed - 6m 2s, and several more followed, but none was a pager or SMS alert. A community report followed at 00:37 UTC and a social media report at 01:13 UTC.

Remediation

• Stopped the osmdbt timer
• Put OpenStreetMap.org into a read-only maintenance window to suppress edits
• Ran osmdbt by hand with fakelog to skip ahead of the PostgreSQL replication condition
• Restarted the osmdbt timer for minutely diffs
• Ended the read-only maintenance window

Action items

• Upgrade PostgreSQL to 17.x
• Raise an issue with the PostgreSQL developers

Lessons

Communication with the osmdbt developer was crucial for working out the workaround, and the operations team's understanding of osmdbt was limited.

Sources

• https://operations.osmfoundation.org/2025/07/11/post-mortem.html · blog · archived copy

References

• https://github.com/openstreetmap/osmdbt/blob/master/man/osmdbt.md#recovery-procedure
• https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=4909b38af0
• https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=d87d07b7ad3b782cb74566cd771ecdb2823adf6a

Related cases

• IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026

  same root cause · shared Database, Database replication

• IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013

  same root cause · shared Database, Database replication

• IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012

  same root cause · shared Database, Database replication

• IL-0451 Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026

  same root cause · shared Database

• IL-0313 Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026

  same root cause · shared Database

• IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025

  same root cause · shared Database

• IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

  same root cause · shared Database

• IL-0436 A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025

  same root cause · shared Database

• IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025

  same root cause · shared Database

• IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databasesUpstash · 2025

  same root cause · shared Database replication

• IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024

  same root cause · shared Database

• IL-0373 Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024

  same root cause · shared Database

• IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

  same root cause · shared Database

• IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023

  same root cause · shared Database

• IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023

  same root cause · shared Database

• IL-0385 GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021

  same root cause · shared Database

• IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021

  same root cause · shared Database

• IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

  same root cause · shared Database

• IL-0486 Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020

  same root cause · shared Database

• IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019

  same root cause · shared Database

• IL-0439 Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018

  same root cause · shared Database

• IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017

  same root cause · shared Database

• IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016

  same root cause · shared Database

• IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014

  same root cause · shared Database

• IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014

  same root cause · shared Database

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0527
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0527/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.