Cases1 case

1 to 1 of 1

1. IL-0516 matrix.org homeserver rooms stopped working with "No create event in auth events" as a cleanup job deleted live state

   Matrix.org Foundation2025

   Symptoms

   Users of the matrix.org homeserver found rooms that simply stopped working: sending a new message or joining as a new member failed, and where an error was shown at all it was "No create event in auth events". The team first heard of it from a user's bug report and similar reports in public Matrix rooms and on social media.

   High error rateData loss

   Trigger

   A background cleanup job that removes state groups which were created but never used began working on the corrupted part of the state_groups_state table; the first errors appeared in the logs on 27 June 2025.

   Root cause

   Data loss or corruption

   Component

   Database

   Contributing factor

   –

   Detection

   A bug report from a user, followed by similar reports in public Matrix rooms and on social media.

   Remediation

   • Disabled the cleanup job that removes redundant state groups as a precaution
   • Restored the 26 June database backup into a new Postgres instance on Amazon EC2
   • Added a Postgres constraint that rejects deleting the state of a state group still in use
   • Rebuilt the corrupted index on state_groups_state overnight
   • Found the state wrongly removed by the cleanup job by comparing the restored backup with production, and re-inserted it

   Lessons

   Index corruption of this kind can sit unnoticed for months or years, because index entries that point into empty table space are ignored; it shows only once Postgres reuses that space. pg_amcheck did not detect it, although its check confirmed that no index rows were missing and the index only had extra ones.

   View case

By root cause1