Cases1 case

1 to 1 of 1

1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks

   OpenStreetMap Foundation2024

   Symptoms

   At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

   Network unreachableService outage

   Trigger

   HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

   Root cause

   Dependency failure

   Component

   ISP and transit networksDatabase replication

   Contributing factor

   Single point of failureNo runbook

   Detection

   StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

   Remediation

   • Activated a read-only backup instance of OpenStreetMap.org in Dublin
   • Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
   • HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed

   Action items

   • Move to a multi-ISP architecture so another provider can take over when one fails
   • Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
   • Add deeper route and latency monitoring and an automated escalation protocol
   • Hold failover drills for switching between ISPs and keep replication up to date
   • Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
   • Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
   • Document outage communication procedures and delegate communication during outages
   • Investigate decoupling the OAuth service so authentication keeps working while the database is read-only

   Lessons

   Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

   View case

By root cause1