Incident Library

IL-0242 Matrix.org Foundation · 2025-09-02database

Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours

The matrix.org homeserver is a main Synapse instance with hundreds of workers on one logical PostgreSQL cluster of two bare-metal servers at Mythic Beasts, each with its own mdraid RAID 10 array. The secondary replicates from the primary through streaming replication, and the primary archives WALs continuously to an S3 bucket in AWS, alongside weekly full backups and daily incremental backups. At the time db-02 was the primary and db-01 the secondary. When the database reached about 51TB, 90% of disk capacity, Mythic Beasts added two NVMe drives to each server on 2 September 2025, and at 11:17 UTC an existing drive disappeared from db-02's array. The team failed over to db-01 at 13:27 UTC, restarted db-02 as a replica, and once it had caught up decided at 13:53 UTC to reboot it to repair the array. db-02 came back in recovery mode because another drive was missing; the RAID headers were gone on both missing drives, so a fresh array was created, and db-02 was back online at 16:11 UTC. WAL archiving from db-01 to S3 was unblocked at 16:50 UTC.

At 17:20 UTC PostgreSQL on the empty db-02 was upgraded to the latest patch version, and at 17:25 UTC the team began restoring it with wal-g. Two failed attempts left directories in the data path, and while removing them the data directory of the primary on db-01 was deleted by mistake, although the on-call group was working as a trio. PostgreSQL on db-01 was left running in case open file handles kept the data on disk. From 17:30 UTC db-02 was restored from the offsite backups in S3, Synapse and the damaged primary were shut down, and db-01's data partition was remounted read-only at 18:06 UTC. The team decided against undeletion: extundelete and ext4magic had been unmaintained for a decade and cannot work on an unmounted filesystem, and the files recovered with R-Linux could not be trusted. The WALs that db-01 had never archived to S3 were copied to db-02 for the replay.

The full weekly backup finished extracting at 07:21 UTC the next morning, but wal-g failed on the incremental backups because of a known bug. The team updated wal-g, patched it to accept a data directory that already held the full base backup so nothing had to be pulled from S3 again, and restored the incremental backups between 09:35 and 10:03 UTC. With a restore_command wrapper that fetched WALs from S3 and from the files copied from db-01, PostgreSQL replayed the WALs in standby mode from 11:00 to 16:27 UTC. db-02 was promoted to primary at 16:45 UTC and Synapse was started at 16:54 UTC; after several overloads as everyone reconnected at once, all workers were online and stable by 18:00 UTC. No data was lost. Later, db-01 failed to boot because it was rebooted while its array was reshaping after the new disks were added; once the array was fixed, db-01 was rebuilt as a replica from a base backup of db-02.

Severity
–
Scope
matrix.org homeserver unavailable from 2025-09-02 17:45 UTC to 2025-09-03 18:00 UTC; no data was lost
Users affected
–
Region
–
Time to detect
–
Time to resolve
24h 15m

Symptoms

On 2 September 2025 the matrix.org homeserver went down at 17:45 UTC and stayed down until 18:00 UTC the next day. Both machines of its PostgreSQL cluster were out of action: the original primary had lost its RAID array during a disk expansion, and the data directory of the new primary had been deleted by mistake.

The trouble began at 11:17 UTC, when one existing drive disappeared from the RAID 10 array of db-02, the primary database server, shortly after the hosting provider Mythic Beasts added two NVMe drives to each server. Monitoring fired and the array kept working in degraded mode. After a planned failover to db-01 at 13:27 UTC, a reboot of db-02 at 14:01 UTC left it in recovery mode, with no network, no ssh and no PostgreSQL, because another drive was now missing and the array could not be assembled. Meanwhile the new primary on db-01 did not archive WALs to S3 because of an issue in the archiving script, and WALs kept piling up on db-01.

While restoring a replica on the rebuilt db-02, the first wal-g backup-fetch run aborted because it was started from inside the PostgreSQL data directory, which made its tee command fail, and the second run failed because the aborted restore had left the data directory non-empty. While clearing those leftovers, the data directory of the primary on db-01 was deleted instead. The partly wiped primary threw errors without crashing Synapse, and the team shut down Synapse and the database.

The restore from S3 then hit more problems. After the full weekly backup had been extracted at 07:21 UTC the next morning, wal-g failed on the first daily incremental backup because of a wal-g bug whose fix had been applied to the disaster recovery test playbook but not to the production servers. PostgreSQL on db-02 then refused to start in standby mode because it could reach no primary and had no restore_command set. Replaying about 18 hours of WALs took 5.5 hours, where the team had hoped for about 10 minutes per hour of WALs.

Symptom Service outageData loss

Root cause

Root cause Operator error

Component DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor Overlapping failuresHarmful manual actionUnpatched known bugNo safeguard on destructive change
Trigger
During maintenance to add drives, an existing drive disappeared from the primary database server's RAID 10 array at 11:17 UTC, which led to a failover and a rebuild of that server. The outage itself began when the data directory of the new primary on db-01 was deleted while clearing a failed restore on db-02.

Detection

Monitoring fired when the drive disappeared from db-02's RAID array at 11:17 UTC, and Mythic Beasts confirmed the issue. The team realised it had deleted db-01's data directory, and the backend team was then paged, an incident was opened and an emergency was declared.

Remediation

Action items

Lessons

Most of the difficulty came from the size of the database: restoring 51TB and replaying WALs took far longer than hoped, and incremental backups restore much faster than WAL replay. Running a destructive command on the wrong server was the turning point, and Matrix.org looks for better tooling and processes rather than blaming the individual; working as a trio did not catch the mistake, and server names that suggest a role caused confusion because a primary and a secondary swap roles. The recovery lost no data, but the recovery time needs to be shorter, and the wal-g versions on the disaster recovery server and the production servers had drifted apart.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database backups and snapshots, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.