Incident Library
Applied filters Root causeFaulty deploymentComponentDatabase backups and snapshotsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026
    Symptoms

    From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

    Data lossResponse quality degradationHigh latency
    Trigger

    On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging
    Detection

    Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

    Remediation
    • Rolled back the compaction change
    • Restored recent data first, rebuilt from the Kafka logs
    • Changed the restoration process to make historical data recovery more efficient
    • Restored historical data from the weekly TSM snapshots
    • Kept the cluster online and restored data incrementally so ingest and queries could continue
    Action items
    • Expand test coverage for changes to critical storage components such as the compactor, with scenarios closer to production-scale workloads
    • Strengthen safeguards around compaction so failures are detected and handled safely
    • Review restoration workflows and procedures to reduce elevated TTBR and slow queries during large restores
    • Revisit the backup and restore strategy to speed up cluster-wide restorations
    View case
  2. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran
    Render2023
    Symptoms

    After Point-in-time Recovery (PITR) was released to eligible new Render Postgres databases on 21 April 2023, 50% of new connections to PITR-enabled databases failed while a PITR backup job was running. A backup takes 15 to 45 minutes depending on database size, and existing connections, such as those held through connection pooling, kept working.

    Intermittent connection failuresMisrouted requests
    Trigger

    Point-in-time Recovery was released to eligible new databases on 21 April 2023, and its backup job carried an incorrect networking configuration.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase backups and snapshotsService discovery
    Contributing factor
    –
    Remediation
    • Disabled backups for affected databases until they could be scheduled in low-traffic hours
    • Changed the backup job's network labels so the routing layer no longer picks it up as a database
    Action items
    • Tighten the process for sharing incident updates so problems are not declared resolved too early
    • Use more specific network labels for the database routing layer so ineligible IPs are never added
    • Run a skipped asynchronous job in the next scheduled window instead of immediately
    • Use open source benchmarking software to reproduce issues customers notice
    View case
Check your cluster for this

These 2 cases involve Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
Database2Database backups and snapshots2Service discovery1
Contributing factor
Missed in review1Unrepresentative staging1Gap in automated tests1Unsafe failure behaviour1
Organisation
InfluxData1Render1
  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026
  2. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran
    Render2023