Cases2 cases

1 to 2 of 2

1. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable

   SourceHut2023

   Symptoms

   At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

   Service outage

   Trigger

   remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

   Root cause

   Hardware failure

   Component

   Database

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

   Remediation

   • Sent SourceHut's emergency on-call contact in the datacenter area to go on site
   • Moved remilia's disks into the chassis of build worker cirno1.sr.ht and brought it online as remilia

   Action items

   • Make the off-site database copy able to serve as a failover database host
   • Re-prioritize the migration to the EU
   • Set up new database failover
   • Add a new EU build worker to backfill build capacity
   • Record a better hardware inventory for future reference

   View case
2. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case

By root cause1