Cases3 cases

1 to 3 of 3

1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry

   Harness2025

   Symptoms

   At 1:47am GMT on 28 October 2025, Harness's automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing, and some requests to http://config.ff.harness.io/client/auth received 502 responses.

   High error rateImage pull failures

   Trigger

   During a routine deployment, Kubernetes rescheduled some pods of a Feature Flags dependency onto new nodes that did not have the image cached locally.

   Root cause

   Data loss or corruption

   Component

   Container registry and its credentials

   Contributing factor

   Cache-masked failure

   Detection

   Automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing at 1:47am GMT.

   Remediation

   • Restored the missing image to Google Artifact Registry
   • Redeployed the impacted pods

   Action items

   • Review and strengthen the image retention policies in Google Artifact Registry to prevent unintended deletions
   • Validate automatically that all production images are kept for a defined minimum period
   • Add monitoring and alerting for image pull errors

   View case
2. IL-0516 matrix.org homeserver rooms stopped working with "No create event in auth events" as a cleanup job deleted live state

   Matrix.org Foundation2025

   Symptoms

   Users of the matrix.org homeserver found rooms that simply stopped working: sending a new message or joining as a new member failed, and where an error was shown at all it was "No create event in auth events". The team first heard of it from a user's bug report and similar reports in public Matrix rooms and on social media.

   High error rateData loss

   Trigger

   A background cleanup job that removes state groups which were created but never used began working on the corrupted part of the state_groups_state table; the first errors appeared in the logs on 27 June 2025.

   Root cause

   Data loss or corruption

   Component

   Database

   Contributing factor

   –

   Detection

   A bug report from a user, followed by similar reports in public Matrix rooms and on social media.

   Remediation

   • Disabled the cleanup job that removes redundant state groups as a precaution
   • Restored the 26 June database backup into a new Postgres instance on Amazon EC2
   • Added a Postgres constraint that rejects deleting the state of a state group still in use
   • Rebuilt the corrupted index on state_groups_state overnight
   • Found the state wrongly removed by the cleanup job by comparing the restored backup with production, and re-inserted it

   Lessons

   Index corruption of this kind can sit unnoticed for months or years, because index entries that point into empty table space are ignored; it shows only once Postgres reuses that space. pg_amcheck did not detect it, although its check confirmed that no index rows were missing and the index only had extra ones.

   View case
3. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates

   BattleMetrics2022

   Symptoms

   At 03:23 UTC on 27 March 2022, BattleMetrics' primary PostgreSQL database, which holds all account, server and player information, went into read-only mode. Users first saw the website fail to load, and soon RCON updates and queries began to fail as well.

   Writes rejectedService outage

   Trigger

   A corruption of the index on the player identifiers table, which BattleMetrics' logs place at about 26 February 2022, made every vacuum job fail, so transaction IDs kept climbing until Postgres stopped accepting writes on 27 March.

   Root cause

   Data loss or corruption

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotOverlapping failuresNo headroom

   Detection

   Users failed to load the website and reported it as their API keys expired, then RCON updates and queries began to fail, and the team found that the main database no longer accepted writes.

   Remediation

   • Shut the database down and dropped the corrupted player identifiers index in single-user mode
   • Ran the vacuum again, which succeeded and let the database accept writes
   • Rebuilt the player identifier index, which took about two hours
   • Shut down the query workers and gave RCON updates priority
   • Pointed services at the new Postgres primary
   • Bootstrapped new replicas one after another to take load off the primary
   • Restarted queries with throttling, prioritised by server population and rank, after RCON updates had caught up

   Action items

   • Monitor errors in the Postgres logs
   • Monitor and alert on transaction IDs
   • Set up a logical replica in addition to physical replication
   • Build a faster network to speed up recovery from failure

   Lessons

   BattleMetrics names skipping health checks on transaction ID size as its critical mistake: the index corruption silently broke vacuum for a month, and without monitoring of transaction IDs nobody saw the database approach wraparound. Because the corruption had been replicated to the replicas, failing over did not help, and the backup servers and downloadable backups carried the same corruption. The team also found that, contrary to the error message, a full vacuum on a server in wraparound protection does not need single-user mode, and chose to run a slow, delayed service over no service while replicas were rebuilt.

   View case

By root cause1