Incident Library
Applied filters Root causeCertificate or credential expirySymptomData lossOrganisationLootLockerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours
    LootLocker2026sev2
    Symptoms

    About an hour after LootLocker cut over from a managed database provider to its own MySQL deployment on 2026-04-28, data anomalies showed up in production. A small window of writes made shortly before the migration, spanning 2026-04-19 to 2026-04-20, had never reached the new database, and the service stayed degraded for about three hours after the cutover.

    Data loss
    Trigger

    On 2026-04-20 at about 16:20 BST, a routine credential rotation, made as part of an unrelated configuration fix, changed the credentials that the temporary replication process used for its database connection, and the process crashed.

    Root cause
    Certificate or credential expiry
    Component
    DatabaseDatabase replication
    Contributing factor
    Change with a delayed effectMonitoring blind spotUnverified assumptionNo safeguard on destructive change
    Detection

    Data anomalies were detected in production at about 11:30 BST, about an hour after the cutover.

    Remediation
    • Restored normal service behaviour first and deferred the root cause analysis
    • Requested a restore of the deleted source database from the previous provider
    • Replayed the affected records for impacted users from the restored source database
    Action items
    • Perform technical changes of comparable risk in a scheduled maintenance window instead of as live cutovers
    • Require a pre-cutover checklist that verifies replication or sync processes are live and caught up just before cutover
    • Require an explicit downstream impact review for credential rotations of service accounts used by long-running processes
    • Keep source systems until integrity in the target has been independently verified after cutover
    • Publish planned maintenance and migration notices in the in-product News feed as well as Discord, with clear impact statements and more advance notice
    Lessons

    The failure was a silent side effect of an unrelated change made eight days earlier: the replication process recovered and kept working, so there was no signal that data had been dropped, and a pre-cutover integrity check would have caught it. Deleting the source database before validating the new one made recovery much harder. Choosing a live migration to avoid downtime left no clean point to verify integrity, and the resulting degradation and two days of piecemeal recovery far exceeded what a short planned maintenance window would have cost. The announcement on Discord was not clear enough about the impact and came with too little notice.

    View case
Check your cluster for this

This case involves Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Certificate or credential expiry1 case

Component
DatabaseDatabase replication
Contributing factor
Unverified assumptionChange with a delayed effectMonitoring blind spotNo safeguard on destructive change
Organisation
LootLocker
  1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours
    LootLocker2026