Cases2 cases

1 to 2 of 2

1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks

   InfluxData2026

   Symptoms

   From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

   Data lossResponse quality degradationHigh latency

   Trigger

   On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging

   Detection

   Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

   Remediation

   • Rolled back the compaction change
   • Restored recent data first, rebuilt from the Kafka logs
   • Changed the restoration process to make historical data recovery more efficient
   • Restored historical data from the weekly TSM snapshots
   • Kept the cluster online and restored data incrementally so ingest and queries could continue

   Action items

   • Expand test coverage for changes to critical storage components such as the compactor, with scenarios closer to production-scale workloads
   • Strengthen safeguards around compaction so failures are detected and handled safely
   • Review restoration workflows and procedures to reduce elevated TTBR and slow queries during large restores
   • Revisit the backup and restore strategy to speed up cluster-wide restorations

   View case
2. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup

   Sakura Internet2012

   Symptoms

   During emergency maintenance on 5 January 2012, a bulk deletion of unneeded file systems on the Sakura Cloud storage device also deleted 53 disks that customers were using. The deleted disks could not be restored from backup.

   Data loss

   Trigger

   In the emergency maintenance on 5 January 2012, Sakura Internet ran a batch command written only for that maintenance to delete unneeded file systems on the storage device in bulk.

   Root cause

   Operator error

   Component

   Cloud block storageDatabase backups and snapshots

   Contributing factor

   Missed in reviewUntested backup

   Action items

   • Simplify the process for deleting unneeded file systems
   • Do not run scripts that have not been reviewed and tested enough

   View case

By root cause2