Cases1 case

1 to 1 of 1

1. IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc error

   OpenStreetMap Foundation2025

   Symptoms

   At 21:40 UTC on 25 June 2025, Alertmanager alerted the OpenStreetMap operations team: minutely replication feed delayed - 6m 2s. Since 21:33 UTC every run of the osmdbt timer that produces the minutely replication diffs had failed with ERROR: invalid memory alloc request size 1243650064.

   Replication lag

   Root cause

   Software defect

   Component

   Database replicationDatabase

   Contributing factor

   Pages not delivered

   Detection

   Alertmanager sent the first alert at 21:40 UTC, minutely replication feed delayed - 6m 2s, and several more followed, but none was a pager or SMS alert. A community report followed at 00:37 UTC and a social media report at 01:13 UTC.

   Remediation

   • Stopped the osmdbt timer
   • Put OpenStreetMap.org into a read-only maintenance window to suppress edits
   • Ran osmdbt by hand with fakelog to skip ahead of the PostgreSQL replication condition
   • Restarted the osmdbt timer for minutely diffs
   • Ended the read-only maintenance window

   Action items

   • Upgrade PostgreSQL to 17.x
   • Raise an issue with the PostgreSQL developers

   Lessons

   Communication with the osmdbt developer was crucial for working out the workaround, and the operations team's understanding of osmdbt was limited.

   View case

By root cause1