IL-0509 SourceHut · 2024-01-10networkingsecurity
SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
SourceHut runs its own servers in colocation at three datacenters: PHL, the primary; FRE, used for off-site backups, including a Postgres standby replicated in real time and daily backups of large data such as git.sr.ht; and AMS, a research installation for a planned migration to the EU, provisioned to run most of a SourceHut installation. In PHL, SourceHut rented network service from its colocation provider, which provisioned a subnet out of its own AS and routed it through Cogent and Level 3. In response to a layer 3 DDoS attack that began at about 06:00 UTC on 10 January 2024, Cogent null-routed SourceHut's downstream AS. Service mostly returned by 18:00 UTC after the null route was narrowed to the one targeted /32, but the attack broadened the next morning and the colocation provider null-routed the whole subnet again. The write-up says the upstream network's response did more damage to SourceHut's infrastructure than anything else.
At 09:00 UTC on 11 January SourceHut decided to migrate to AMS in an emergency and restore from backups. The standby database was less than 30 seconds behind production and large storage backups were no more than 12 hours old. The team built a new SourceHut installation from scratch: it moved DNS to an external provider, brought up a new primary database, restored backups from FRE to AMS, applied the difference between production data and the last backups by hand, provisioned a new mail system, and brought services up as their restores finished. A layer 3 attack can only be absorbed with more bandwidth than the attacker or filtered far enough upstream, and many DDoS protection services only cover web traffic, while SourceHut also serves SSH, SMTP and IRC and terminates end-to-end encryption on its own servers. Cloudflare's quote was beyond SourceHut's means, so the team set up an OVH box that NATs traffic through OVH's DDoS-resistant network to a production subnet in AMS kept secret. From 12 January a temporary subnet from the PHL provider gave out-of-band access to the PHL servers.
Services came back in stages. paste and the project hub, which need only a working SQL server, returned at about 19:00 UTC on 11 January; git.sr.ht came back read-only from somewhat stale backups by 20:00 UTC and read/write at 11:00 UTC on 13 January; hg.sr.ht at 08:20 UTC on 15 January; builds.sr.ht at 18:00 UTC on 16 January on dedicated OVH bare-metal servers, because AMS lacked compute; and chat.sr.ht, which needs a separate IPv6 address per user for outbound IRC connections, last at 10:30 UTC on 17 January. pages.sr.ht used S3-compatible object storage that the off-site backups did not cover, so it ran over an out-of-band link to PHL while the slow object storage migration continued, and users whose apex records pointed directly at the PHL subnet had to change them. The outage lasted 170 hours.
- Severity
- –
- Scope
- All SourceHut services; partial service returned from 12 January and full service on 17 January, and pages.sr.ht users with apex records on the PHL subnet had to update them
- Users affected
- all SourceHut users; paying users were credited 7 days of free service
- Region
- PHL datacenter
- Time to detect
- 0m
- Time to resolve
- 170h
Symptoms
At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.
The PHL network was unreachable because Cogent, one of the two carriers through which the colocation provider routed SourceHut's subnet, announced null routes for SourceHut's downstream AS. SourceHut could not mitigate a layer 3 attack without its network provider, but the ticketing portal it used to page the provider had been deprecated after the provider was acquired twice, and SourceHut's account had not been moved to the new system. The provider could not be reached until its office hours began at 14:00 UTC.
Once reached, the provider got Cogent to narrow the null route from SourceHut's entire /24 to the single /32 address being targeted, and service was mostly back by 18:00 UTC. At about 06:30 UTC the next morning the attack escalated to other parts of the PHL subnet, the colocation provider null-routed the subnet again, and it stayed unreachable from then on.
The attacker targeted SourceHut specifically and followed it, moving the attack to new infrastructure as it was being set up. When SourceHut first NAT'd all traffic from an OVH box through to AMS, the attack on the OVH infrastructure was briefly forwarded to AMS before OVH's mitigations started, which made the new OVH account look like the source of an outgoing DDoS and took some work to resolve with OVH.
Root cause
Root cause Dependency failure
- Trigger
- A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.
Detection
SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.
Remediation
- Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
- Moved DNS to an external DNS provider
- Brought up a new primary database in AMS
- Restored backups from FRE to AMS
- Applied the difference between production data and the last backups by hand
- Provisioned a new mail system and configured the mail services for it
- Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
- Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
- Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
- Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
- Ran build workers on dedicated bare-metal OVH servers
Action items
- Provision a new standby database
- Configure backups from AMS to FRE
- Install a new monitoring system
- Retire the PHL datacenter installation and ship its servers to AMS
- Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
- Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
- Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
Lessons
The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.
Sources
Related cases
IL-0526OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinksOpenStreetMap Foundation · 2024same root cause · shared ISP and transit networks
IL-0259Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customersFastmail · 2023same root cause · shared ISP and transit networks
IL-0267Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were downWikimedia Foundation · 2023same root cause · shared BGP routing
IL-0255Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020same root cause · shared BGP routing
IL-0358Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed trafficPacket · 2019same root cause · shared BGP routing
IL-0258FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went downFastmail · 2016same root cause · shared BGP routing
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0509
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0509/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.