Cases2 cases

1 to 2 of 2

1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry

   Harness2025

   Symptoms

   At 1:47am GMT on 28 October 2025, Harness's automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing, and some requests to http://config.ff.harness.io/client/auth received 502 responses.

   High error rateImage pull failures

   Trigger

   During a routine deployment, Kubernetes rescheduled some pods of a Feature Flags dependency onto new nodes that did not have the image cached locally.

   Root cause

   Data loss or corruption

   Component

   Container registry and its credentials

   Contributing factor

   Cache-masked failure

   Detection

   Automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing at 1:47am GMT.

   Remediation

   • Restored the missing image to Google Artifact Registry
   • Redeployed the impacted pods

   Action items

   • Review and strengthen the image retention policies in Google Artifact Registry to prevent unintended deletions
   • Validate automatically that all production images are kept for a defined minimum period
   • Add monitoring and alerting for image pull errors

   View case
2. IL-0516 matrix.org homeserver rooms stopped working with "No create event in auth events" as a cleanup job deleted live state

   Matrix.org Foundation2025

   Symptoms

   Users of the matrix.org homeserver found rooms that simply stopped working: sending a new message or joining as a new member failed, and where an error was shown at all it was "No create event in auth events". The team first heard of it from a user's bug report and similar reports in public Matrix rooms and on social media.

   High error rateData loss

   Trigger

   A background cleanup job that removes state groups which were created but never used began working on the corrupted part of the state_groups_state table; the first errors appeared in the logs on 27 June 2025.

   Root cause

   Data loss or corruption

   Component

   Database

   Contributing factor

   –

   Detection

   A bug report from a user, followed by similar reports in public Matrix rooms and on social media.

   Remediation

   • Disabled the cleanup job that removes redundant state groups as a precaution
   • Restored the 26 June database backup into a new Postgres instance on Amazon EC2
   • Added a Postgres constraint that rejects deleting the state of a state group still in use
   • Rebuilt the corrupted index on state_groups_state overnight
   • Found the state wrongly removed by the cleanup job by comparing the restored backup with production, and re-inserted it

   Lessons

   Index corruption of this kind can sit unnoticed for months or years, because index entries that point into empty table space are ignored; it shows only once Postgres reuses that space. pg_amcheck did not detect it, although its check confirmed that no index rows were missing and the index only had extra ones.

   View case

By root cause1