Incident Library

IL-0380 InfluxData · 2026-02-09databaseaws

A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks

Data in InfluxDB Cloud 2 is split into 64 partitions, and each partition is compacted regularly because uncompacted data is slower to query. In January, partition 47 of prod101-us-east-1 stopped compacting successfully, partially compacted files piled up in it and performance suffered. InfluxData built an optimisation that runs several compaction jobs on a partition in parallel, each on a subset of the data by org id prefix, and two more engineers reviewed it in the standard release process. Under certain circumstances the new code made the compactor fail to read some source TSM data files. The compactor did not handle the failure: it silently skipped those TSM files, compacted the rest and replaced the original files with the result. Code review and internal testing did not catch this, and it appeared only after the change reached prod101-us-east-1 on 9 February.

On 10 February, after more tickets, InfluxData found the cause, rolled back the compaction change and began restoring data. Each Cloud 2 cluster has weekly TSM snapshots, and all incoming writes are also recorded as snapshots in Kafka logs; together these hold a complete record of the written data. Only partitions compacted while the faulty code ran were affected, but as a precaution the restore took recent data first, rebuilt from the Kafka logs, and historical data afterwards from the TSM snapshots. Because customers use the cluster for real-time work, it stayed online and the restore ran incrementally so ingest and queries could continue, which took longer than an offline restore from a full backup and at times raised TTBR and slowed queries. The restore was done first for a test set of customers, the process was changed on 17 February to speed up historical recovery, recent data was back for everyone on 23 February and historical data for all paying customers on 8 March. The incident was marked resolved on 10 March 2026.

Severity
–
Scope
Data in partitions compacted while the faulty code ran on prod101-us-east-1 was missing from queries until restored from backup; elevated TTBR and slower queries during the restore; no permanent data loss
Users affected
–
Region
AWS us-east-1
Time to detect
–
Time to resolve
–

Symptoms

From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

The missing data was still in backups, so nothing was lost for good, but bringing it back took weeks. Recent data from 2 to 9 February was restored for all customers on 23 February, and historical data from before 2 February for all paying customers on 8 March. During the restore some customers saw elevated time-to-become-readable (TTBR) and slower queries.

Symptom Data lossResponse quality degradationHigh latency

Root cause

Root cause Faulty deployment

Component DatabaseDatabase backups and snapshots
Contributing factor Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging
Trigger
On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

Detection

Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database backups and snapshots among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.