Cases2 cases

1 to 2 of 2

1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours

   LootLocker2026sev2

   Symptoms

   About an hour after LootLocker cut over from a managed database provider to its own MySQL deployment on 2026-04-28, data anomalies showed up in production. A small window of writes made shortly before the migration, spanning 2026-04-19 to 2026-04-20, had never reached the new database, and the service stayed degraded for about three hours after the cutover.

   Data loss

   Trigger

   On 2026-04-20 at about 16:20 BST, a routine credential rotation, made as part of an unrelated configuration fix, changed the credentials that the temporary replication process used for its database connection, and the process crashed.

   Root cause

   Certificate or credential expiry

   Component

   DatabaseDatabase replication

   Contributing factor

   Change with a delayed effectMonitoring blind spotUnverified assumptionNo safeguard on destructive change

   Detection

   Data anomalies were detected in production at about 11:30 BST, about an hour after the cutover.

   Remediation

   • Restored normal service behaviour first and deferred the root cause analysis
   • Requested a restore of the deleted source database from the previous provider
   • Replayed the affected records for impacted users from the restored source database

   Action items

   • Perform technical changes of comparable risk in a scheduled maintenance window instead of as live cutovers
   • Require a pre-cutover checklist that verifies replication or sync processes are live and caught up just before cutover
   • Require an explicit downstream impact review for credential rotations of service accounts used by long-running processes
   • Keep source systems until integrity in the target has been independently verified after cutover
   • Publish planned maintenance and migration notices in the in-product News feed as well as Discord, with clear impact statements and more advance notice

   Lessons

   The failure was a silent side effect of an unrelated change made eight days earlier: the replication process recovered and kept working, so there was no signal that data had been dropped, and a pre-cutover integrity check would have caught it. Deleting the source database before validating the new one made recovery much harder. Choosing a live migration to avoid downtime left no clean point to verify integrity, and the resulting degradation and two days of piecemeal recovery far exceeded what a short planned maintenance window would have cost. The announcement on Discord was not clear enough about the impact and came with too little notice.

   View case
2. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days

   PostHog2026

   Symptoms

   On 19 February 2026, a node of the ClickHouse cluster behind PostHog's Logs product crashed, restarted and could not start again because it could not find a large amount of its data. Engineers then found that almost all of the Logs data stored on S3 was gone, so logs collected in the US region more than 3 days earlier were lost. Only the Logs product was affected.

   Data loss

   Trigger

   At 10:54 UTC a routine mutation was run to materialise an index. When it finished at 11:02, it set off a bug in ClickHouse's zero-copy replication.

   Root cause

   Software defect

   Component

   DatabaseDatabase replicationObject storageDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotSingle point of failureUntested backup

   Detection

   No automated alarm fired. At 19:40 UTC a node in the cluster crashed, restarted and failed to start because of the missing data, and the engineers who investigated found that most of the S3-backed data was gone.

   Remediation

   • Switched Logs to a new table without zero-copy replication and caught up on recent messages
   • Backfilled the Logs data from the 3 days of Kafka history

   Action items

   • Disable zero-copy replication on all clusters
   • Audit the whole Logs ClickHouse configuration and keep experimental features out of production
   • Enable S3 versioning on the storage buckets so objects deleted by a database bug can be recovered
   • Spot-check configurations and data integrity strategies for single points of failure before a product becomes generally available

   Lessons

   Running Logs in its own cluster kept every other product unaffected, and 3 days of Kafka retention saved the recent data. An experimental setting chosen for an early internal stage was allowed to stay in production. Monitoring covered only incoming and recent data, so petabytes were deleted over 8 hours without an alarm. Relying on database replication alone over shared storage was a single point of failure, and S3 versioning would have let the deleted files be recovered.

   View case

By root cause2