Cases2 cases

1 to 2 of 2

1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks

   InfluxData2026

   Symptoms

   From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

   Data lossResponse quality degradationHigh latency

   Trigger

   On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging

   Detection

   Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

   Remediation

   • Rolled back the compaction change
   • Restored recent data first, rebuilt from the Kafka logs
   • Changed the restoration process to make historical data recovery more efficient
   • Restored historical data from the weekly TSM snapshots
   • Kept the cluster online and restored data incrementally so ingest and queries could continue

   Action items

   • Expand test coverage for changes to critical storage components such as the compactor, with scenarios closer to production-scale workloads
   • Strengthen safeguards around compaction so failures are detected and handled safely
   • Review restoration workflows and procedures to reduce elevated TTBR and slow queries during large restores
   • Revisit the backup and restore strategy to speed up cluster-wide restorations

   View case
2. IL-0547 Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an index

   Buildkite2025

   Symptoms

   From 04:48 UTC on 10 November 2025, queries on the table that stores annotations in Buildkite Pipelines timed out, CPU load on the affected databases rose sharply, and the Agent API, the REST API, GraphQL and the Web UI slowed down and returned more errors.

   Intermittent timeoutsDatabase overloadHigh error rateHigh latency

   Trigger

   At 04:48 UTC a database migration removed an index on the annotations table of the Pipelines product.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Missed in reviewNo staged rollout

   Detection

   At 04:53 UTC the impact of the removed index was detected as queries timing out on the annotations table.

   Remediation

   • Recreated the removed index, finishing on about half of the database shards by 05:30 UTC
   • Temporarily revoked the application user's permission to read from and write to the annotations table
   • Increased max_parallel_maintenance_workers in Postgres to speed up index creation
   • Finished recreating the index on the remaining shards between 07:50 and 08:43 UTC
   • Freed the agents and cleaned up the job states left inconsistent, handling by hand the cases the system could not self-heal, by 11:56 UTC

   Action items

   • Extend the internal tooling that flags risky migrations in code review to cover index removals and ask engineers to verify an index is no longer needed
   • Roll out high-risk migrations to shards in phases instead of to all shards at once
   • Evaluate tools such as HypoPG to analyse index creation and removal
   • Show index usage data from pganalyze more prominently in the development workflow
   • Automate the manual cleanup of inconsistent agent states so the system can heal itself

   Lessons

   Buildkite said the incident was mainly caused by a database migration that exposed gaps in its review process.

   View case

By root cause1