Incident Library

IL-0516 Matrix.org Foundation · 2025-06-27database

matrix.org homeserver rooms stopped working with "No create event in auth events" as a cleanup job deleted live state

The matrix.org homeserver runs Synapse on a large PostgreSQL database. Synapse stores room state as state groups, mostly as deltas from an earlier state group, in the tables event_to_state_groups, state_groups_state and state_group_edges. Some time after January 2021 and before the end of June 2025, parts of an index on state_groups_state became corrupted: index entries pointed to places in the table data, the heap, where rows had been removed, most likely by the state compressor. Postgres ignores index entries that point into empty space, so for a long time nothing showed. Once new rows were written into that space, as on 3 July for state group 1034753774, the old entries pointed at unrelated live rows. The cleanup job, correctly deleting an unused state group, then also deleted the state of state groups in completely unrelated rooms, and those rooms broke.

The team disabled the cleanup job as a precaution, started restoring the 26 June backup into a new Postgres instance on Amazon EC2, which took several hours, and added a constraint that rejects deleting state from a state group still in use. The warm standby, kept up to date with streaming replication, had the same corruption, and so did the restored backup. A sampling script found millions of affected state groups, all in the range 147M - 541M. The team rebuilt the index, which takes up over 4 TB, overnight, then compared the range the cleanup job had worked on in the restored backup and in production to find the state that had been wrongly removed, and re-inserted it. The scripts started on 5 July, and on 11 July the team announced that the majority of rooms were repaired.

The cause of the index corruption is not known. Synapse and PgCat were ruled out because an application cannot corrupt the inside of a Postgres database, and no fix in the Postgres release notes since 10.12, the version in use in January 2021, matched the pattern. The team considers a hardware problem the most plausible cause and mentions a disk failure on the primary database server in October 2021 that led to a failover to the secondary. The current servers run ext4 on stock Debian kernels with Linux kernel 6.1 and eight 15TB Intel NVME SSDs in RAID10, and fsck, smartctl and mdadm showed no errors.

Severity
–
Scope
Rooms on the matrix.org homeserver whose state groups lost data: sending messages and joining failed
Users affected
–
Region
–
Time to detect
–
Time to resolve
14d

Symptoms

Users of the matrix.org homeserver found rooms that simply stopped working: sending a new message or joining as a new member failed, and where an error was shown at all it was "No create event in auth events". The team first heard of it from a user's bug report and similar reports in public Matrix rooms and on social media.

In affected rooms, some state groups were completely empty and did not even contain the room's create event, which is always part of a room's state, so Synapse rejected new events. A search of the logs of one Synapse process found 0 results for "No create event" from 24 to 26 June 2025, then 48 on 27 June, 1100 on 28 June, 3610 on 29 June and 6902 on 30 June. State groups that had been fine earlier, for example on 29 June, were later corrupt, so something was corrupting existing state groups rather than creating bad ones.

No recent changes were found in the code that modifies state groups, and the state compressor was ruled out because it was not running on matrix.org. Rolling back Synapse was considered but not done, because nobody knew how far back to go. After a Postgres constraint was added to stop deleting state still in use, re-enabling the cleanup job hit it at once: "ERROR: Deleting state_groups_state row when it still exists in state_groups_edges: prev_state_group = 963361509", while the Synapse logs showed it deleting different state groups. tcpdump showed DELETE queries but none for that state group, and suspicion moved between Synapse, the Python Postgres driver, PgCat and Postgres itself.

Reading one state group directly gave the answer: a SELECT on state_groups_state for state_group 483128098 returned a row for state group 963361875 in an unrelated room, while a query Postgres could answer from the index alone returned plausible rows. pg_amcheck on the affected index reported no problems.

Symptom High error rateData loss

Root cause

Root cause Data loss or corruption

Component Database
Contributing factor –
Trigger
A background cleanup job that removes state groups which were created but never used began working on the corrupted part of the state_groups_state table; the first errors appeared in the logs on 27 June 2025.

Detection

A bug report from a user, followed by similar reports in public Matrix rooms and on social media.

Remediation

Action items

–

Lessons

Index corruption of this kind can sit unnoticed for months or years, because index entries that point into empty table space are ignored; it shows only once Postgres reuses that space. pg_amcheck did not detect it, although its check confirmed that no index rows were missing and the index only had extra ones.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.