Incident Library
Applied filters SymptomNetwork unreachableFactorUntested backupClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
    Symptoms

    At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

    Network unreachableService outage
    Trigger

    A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

    Root cause
    Dependency failure
    Component
    ISP and transit networksBGP routing
    Contributing factor
    Single point of failureLimited debug accessUntested backup
    Detection

    SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

    Remediation
    • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
    • Moved DNS to an external DNS provider
    • Brought up a new primary database in AMS
    • Restored backups from FRE to AMS
    • Applied the difference between production data and the last backups by hand
    • Provisioned a new mail system and configured the mail services for it
    • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
    • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
    • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
    • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
    • Ran build workers on dedicated bare-metal OVH servers
    Action items
    • Provision a new standby database
    • Configure backups from AMS to FRE
    • Install a new monitoring system
    • Retire the PHL datacenter installation and ship its servers to AMS
    • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
    • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
    • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
    Lessons

    The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

    View case
Check your cluster for this

This case involves BGP routing and ISP and transit networks. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
BGP routingISP and transit networks
Contributing factor
Untested backupLimited debug accessSingle point of failure
Organisation
SourceHut
  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024