IL-0490 LootLocker · 2026-04-28database

Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours

LootLocker was moving from a large managed database provider to a custom MySQL deployment run with a European infrastructure partner, using a temporary one-off replication process for the length of the migration. On 2026-04-20 a credential rotation crashed that process. It recovered automatically within minutes, but the data batches in flight at the moment of the crash were dropped and never replayed, and because of how replication offsets and timezones interacted, the lost window also included some data from the previous day. Nothing monitored the health or replication lag of the sync process, and no pre-cutover step required a sync health and integrity check.

The cutover ran at 10:34 BST on 2026-04-28 without problems, and at about 10:40 the source database was deleted on the assumption that no rollback path was needed. Data anomalies were found at about 11:30, normal service behaviour was back by about 13:30, and the root cause was identified at about 19:00. Because the source database was gone, LootLocker had to ask the previous provider to restore it before the dropped batches could be replayed, which delayed recovery. The affected records were replayed for the impacted users over 2026-04-29 and 2026-04-30 without further disruption. LootLocker cannot fully rule out that more data written during the window of instability was affected.

Severity: sev2
Scope: LootLocker service degraded for about three hours on migration day; a window of writes from 2026-04-19 to 2026-04-20 was missing from the new database until it was replayed over the next two days
Users affected: a specific subset of users
Region: –
Time to detect: 56m
Time to resolve: 2h 56m

Symptoms

About an hour after LootLocker cut over from a managed database provider to its own MySQL deployment on 2026-04-28, data anomalies showed up in production. A small window of writes made shortly before the migration, spanning 2026-04-19 to 2026-04-20, had never reached the new database, and the service stayed degraded for about three hours after the cutover.

The gap traced back eight days, to a brief crash of the temporary replication process that synced the two databases. The process recovered on its own within minutes and kept replicating normally, so nothing showed at the time that the batches in flight during the crash had been dropped. The team first restored normal service behaviour and put off the root cause analysis, which identified the dropped batches later that evening.

Symptom Data loss

Root cause

Root cause Certificate or credential expiry

Component Database Database replication

Contributing factor Change with a delayed effect Monitoring blind spot Unverified assumption No safeguard on destructive change

Trigger: On 2026-04-20 at about 16:20 BST, a routine credential rotation, made as part of an unrelated configuration fix, changed the credentials that the temporary replication process used for its database connection, and the process crashed.

Detection

Data anomalies were detected in production at about 11:30 BST, about an hour after the cutover.

Remediation

Restored normal service behaviour first and deferred the root cause analysis
Requested a restore of the deleted source database from the previous provider
Replayed the affected records for impacted users from the restored source database

Action items

Perform technical changes of comparable risk in a scheduled maintenance window instead of as live cutovers
Require a pre-cutover checklist that verifies replication or sync processes are live and caught up just before cutover
Require an explicit downstream impact review for credential rotations of service accounts used by long-running processes
Keep source systems until integrity in the target has been independently verified after cutover
Publish planned maintenance and migration notices in the in-product News feed as well as Discord, with clear impact statements and more advance notice

Lessons

The failure was a silent side effect of an unrelated change made eight days earlier: the replication process recovered and kept working, so there was no signal that data had been dropped, and a pre-cutover integrity check would have caught it. Deleting the source database before validating the new one made recovery much harder. Choosing a live migration to avoid downtime left no clean point to verify integrity, and the resulting degradation and two days of piecemeal recovery far exceeded what a short planned maintenance window would have cost. The announcement on Discord was not clear enough about the impact and came with too little notice.

Sources

https://lootlocker.com/blog/db-postmortem-data-loss-during-database-migration/ · blog · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0490
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0490/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.