Cases2 cases

1 to 2 of 2

1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks

   InfluxData2026

   Symptoms

   From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

   Data lossResponse quality degradationHigh latency

   Trigger

   On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging

   Detection

   Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

   Remediation

   • Rolled back the compaction change
   • Restored recent data first, rebuilt from the Kafka logs
   • Changed the restoration process to make historical data recovery more efficient
   • Restored historical data from the weekly TSM snapshots
   • Kept the cluster online and restored data incrementally so ingest and queries could continue

   Action items

   • Expand test coverage for changes to critical storage components such as the compactor, with scenarios closer to production-scale workloads
   • Strengthen safeguards around compaction so failures are detected and handled safely
   • Review restoration workflows and procedures to reduce elevated TTBR and slow queries during large restores
   • Revisit the backup and restore strategy to speed up cluster-wide restorations

   View case
2. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case

By root cause2