Incident Library

IL-0508 SourceHut · 2023-01-11database

All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable

remilia.sr.ht, SourceHut's main database server and a physical host in a US datacenter, suffered a hardware failure at about 13:38 UTC on 11 January 2023 and became unreachable, and the loss of database connectivity took down all services. The off-site database backup was working, but it could not have served as a failover database host. Nobody at the datacenter answered the phone before official hours, so an operator sent SourceHut's emergency contact in the area, who was an hour away. Only one of SourceHut's three staff had a valid passport in case serious work had to be done on site in the US.

While waiting, the team planned to diagnose the server and, if it could not be recovered, move its hard drives into another host and bring that host up as a clone of the database server. It chose cirno1.sr.ht, a build worker redundant with cirno2.sr.ht, so taking it out of service would not affect availability. Datacenter staff power-cycled remilia, which did not come back but suggested the disks were not at fault, and at 15:03 the hardware other than the disks was judged likely lost. The emergency contact arrived at 15:39, confirmed the diagnosis and moved the disks into cirno1's chassis, which brought remilia back in its old state; there was some initial confusion about remilia's disk layout. Prometheus scraped remilia again at 16:04, services began coming back at 16:06, and all services were online at 16:09 UTC, with builds.sr.ht running at reduced build capacity without cirno1.

Severity
–
Scope
All SourceHut services were down; afterwards builds.sr.ht ran with reduced build capacity
Users affected
–
Region
US datacenter
Time to detect
2m
Time to resolve
2h 31m

Symptoms

At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

Prometheus had started failing to scrape metrics from remilia at 13:38. Without database connectivity, services went down across SourceHut. At 13:44 Postfix was shut down so the mail delivery pipeline would not spill over, and Alertmanager was disabled to silence the many alarms.

With no remote way to reach the server, someone had to go to the datacenter. A call to the datacenter at 8:45 AM local time went unanswered because its daytime availability starts at 9 AM, and SourceHut's own emergency contact in the area was an hour's travel away. When datacenter staff power-cycled the server it did not come back and stayed unresponsive.

Symptom Service outage

Root cause

Root cause Hardware failure

Component Database
Contributing factor Single point of failureLimited debug access
Trigger
remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

Detection

Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.