IL-0525 OpenStreetMap Foundation · 2022-06-11networking

OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

OpenStreetMap.org is hosted mainly at the Equinix AM6 data centre in Amsterdam, with failover hosting at Equinix DB4 in Dublin. In Amsterdam a highly available pair of Cisco SG550X-48 switches gets redundant power from a Cisco RPS2300 Redundant Power System fed by power feeds A and B; Switch-1 also has an internal power supply on feed A and Switch-2 one on feed B. The site's single fibre uplink to Cogent runs from Switch-1, a known single point of failure, with a spare fibre module pre-installed in Switch-2 so remote hands can move the uplink by hand. Out-of-band access had been installed in Dublin in October 2021 but not yet in Amsterdam.

On 11 June 2022 at 20:00 UTC Equinix started an 8-hour maintenance of power feed A, during which no OpenStreetMap.org outage had been expected. At 20:18 UTC Switch-1 lost power completely, the uplink to Cogent went down and every OpenStreetMap.org service in Amsterdam lost Internet connectivity, while Switch-2 stayed up on its internal supply from feed B. Without out-of-band access the team could not investigate; it chose not to fail over to Dublin or move the uplink, and waited. The outage ended at 03:45 UTC on 12 June, before the maintenance window closed at 04:00 UTC.

The first hypothesis was that the Redundant Power System's supply on feed B had failed. A site visit found both of its power supplies fully working, and a test on 22 June showed that the Redundant Power System can be in a "standby" state in which it supplies no power to the switches, so a switch fails when its internal supply loses power. How it entered that state is unclear. The power failure scenario had been tested and confirmed working when the Redundant Power System was installed in December 2018. The state can be read on the switches with the CLI command show system, which reports the redundant power supply as "Not Connected" in standby and "Available" when active.

Severity: –
Scope: All OpenStreetMap.org services hosted in Amsterdam (www, API, wiki, planet) had no Internet connectivity for 7 hours 27 minutes
Users affected: –
Region: Amsterdam (Equinix AM6)
Time to detect: –
Time to resolve: 7h 27m

Symptoms

From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

The operations team could not tell why connectivity had failed. The two network switches were meant to keep running on power feed B through a Cisco Redundant Power System. With no out-of-band access equipment installed in Amsterdam, the team had no way to reach the site and investigate while its only uplink was down.

The team decided not to fail over to the Dublin site for what it expected to be the length of the maintenance window, and not to ask Equinix remote hands to move the fibre uplink to the second switch, and waited for the maintenance window to end. The site came back at 03:45 UTC on 12 June, after 7 hours 27 minutes, the longest outage of OpenStreetMap.org in the last 5 years. At 04:22 UTC the team confirmed that Switch-1 had been offline during the power maintenance.

Symptom Network unreachable Service outage

Root cause

Root cause Hardware failure

Component Network switch Data center power and cooling

Contributing factor Single point of failure Limited debug access Monitoring blind spot

Trigger: Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

Detection

–

Remediation

Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

Action items

Acknowledge ongoing outages faster
Add monitoring and alerting of switch PSU and FAN states
Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
Install out-of-band access equipment with dual power feeds in Amsterdam
Install redundant uplinks
Make site failover easier, less manual and lower risk
Introduce an on-call pager system for the operations SRE
Expand the SRE team
Continue and formalise fail-over tests for power, network, upstream link and service outages

Lessons

–

Sources

https://operations.osmfoundation.org/2022/07/20/post-mortem.html · blog · archived copy

References

Related cases

IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024
same root cause · shared Data center power and cooling

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network switch, Data center power and cooling among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0525
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0525/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.