Incident Library

Organisation

Matrix.org Foundation

Matrix.org Foundation · matrix-org-foundation

2 cases · 2025

Root cause Data loss or corruption1Operator error1
Component Database2Database backups and snapshots1Database replication1
Contributing factor Unpatched known bug1Harmful manual action1No safeguard on destructive change1Overlapping failures1
Symptom Data loss2High error rate1Service outage1

Cases

  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025

    Root cause Operator error

  2. IL-0516 matrix.org homeserver rooms stopped working with "No create event in auth events" as a cleanup job deleted live stateMatrix.org Foundation · 2025

    Root cause Data loss or corruption