Incident Library
Applied filters Root causeHardware failureOrganisationSourceHutClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable
    SourceHut2023
    Symptoms

    At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

    Service outage
    Trigger

    remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

    Root cause
    Hardware failure
    Component
    Database
    Contributing factor
    Single point of failureLimited debug access
    Detection

    Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

    Remediation
    • Sent SourceHut's emergency on-call contact in the datacenter area to go on site
    • Moved remilia's disks into the chassis of build worker cirno1.sr.ht and brought it online as remilia
    Action items
    • Make the off-site database copy able to serve as a failover database host
    • Re-prioritize the migration to the EU
    • Set up new database failover
    • Add a new EU build worker to backfill build capacity
    • Record a better hardware inventory for future reference
    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
Database
Contributing factor
Limited debug accessSingle point of failure
Organisation
SourceHut
  1. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable
    SourceHut2023