IL-0526 OpenStreetMap Foundation · 2024-12-15networking

OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks

OpenStreetMap.org's primary servers in Amsterdam reached the Internet only through HE.net, over two redundant, diverse fibre links, and no second ISP was fully operational. At 03:53 UTC on 15 December 2024 HE.net's routing equipment in Amsterdam failed and both links went offline, taking the website and the API offline. StatusCake alerted the team at 04:01 UTC. HE.net had to replace its routing hardware with equipment shipped from California, and gave no estimate of recovery.

After a complete outage of 8 hours the team activated a read-only backup instance in Dublin at 11:31 UTC. Logins to OpenStreetMap.org and services that use its OAuth failed while it was read-only, because OAuth needs database writes to store access tokens. A full failover to Dublin was not done: replication was incomplete, the procedure, written down on 16 December, was complex and seen as risky, and the team decided to wait for Amsterdam connectivity. Equinix escalated the installation of new Equinix Internet connections in Amsterdam and Dublin, which were tested on 16 December. The new Amsterdam link restored full functionality, including map edits, at 12:21 UTC on 17 December, about 56 hours after the start. HE.net restored its service at 00:29 UTC on 18 December, about 68 hours after the start, and routing through HE.net returned to normal.

Severity: –
Scope: OpenStreetMap.org website and mapping API completely offline for 8 hours, then read-only with no edits and no logins (also for community.openstreetmap.org, umap and other services) until 12:21 UTC on 17 December; dev server, Taginfo and some aerial imagery unavailable; tile rendering and Nominatim at reduced capacity
Users affected: –
Region: Amsterdam
Time to detect: 8m
Time to resolve: 2d 8h 28m

Symptoms

At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

By 04:06 UTC the operations team had established that the problem lay with the external ISP, HE.net: both of OpenStreetMap's redundant, diverse fibre links to HE.net in Amsterdam were offline because HE.net's routing equipment there had failed. HE.net confirmed by email that there was no scheduled maintenance and said by telephone that the outage was in Amsterdam, with no estimated time to recover while it waited for Equinix remote hands. Repeated requests over the following days brought no estimate.

At 11:31 UTC a read-only backup instance hosted in Dublin brought the site back with limited functionality and a maintenance notice that edits could not be saved. Because the OAuth service has to write access tokens to the database, users could not log in to OpenStreetMap.org, community.openstreetmap.org, umap and other services while the site was read-only. The dev server, Taginfo and some aerial imagery services in Amsterdam stayed unavailable, and the tile rendering service and Nominatim geocoding ran at reduced capacity.

A full failover to Dublin, covering Postgres and osmdbt, was documented on 16 December but postponed because replication was incomplete and the operation was complex and seen as risky. Map edits came back only at 12:21 UTC on 17 December, over a new Equinix Internet link in Amsterdam.

Symptom Network unreachable Service outage

Root cause

Root cause Dependency failure

Component ISP and transit networks Database replication

Contributing factor Single point of failure No runbook

Trigger: HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

Detection

StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

Remediation

Activated a read-only backup instance of OpenStreetMap.org in Dublin
Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed

Action items

Move to a multi-ISP architecture so another provider can take over when one fails
Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
Add deeper route and latency monitoring and an automated escalation protocol
Hold failover drills for switching between ISPs and keep replication up to date
Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
Document outage communication procedures and delegate communication during outages
Investigate decoupling the OAuth service so authentication keeps working while the database is read-only

Lessons

Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

Sources

https://operations.osmfoundation.org/2025/02/15/post-mortem.html · blog · archived copy

References

https://github.com/openstreetmap/operations/issues/1197

Related cases

IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnetSourceHut · 2024
same root cause · shared ISP and transit networks
IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customersFastmail · 2023
same root cause · shared ISP and transit networks
IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020
same root cause · shared Database replication

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, ISP and transit networks, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0526
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0526/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.