Incident Library
Applied filters ComponentDatabaseSymptomHigh latencyFactorGap in automated testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026
    Symptoms

    From 9 February 2026, InfluxDB Cloud customers on the AWS us-east-1 cluster prod101-us-east-1 found that some recent data did not show up in query responses. The first support ticket about data availability came in on 9 February, the day a compaction change was deployed, and more tickets followed on 10 February.

    Data lossResponse quality degradationHigh latency
    Trigger

    On 9 February 2026 InfluxData deployed to prod101-us-east-1 a compaction change that runs several compaction jobs in parallel on one partition, each on a subset of the data chosen by org id prefix.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unsafe failure behaviourMissed in reviewGap in automated testsUnrepresentative staging
    Detection

    Tickets about data availability were raised, the first on 9 February and more on 10 February, when the status page reported that some recent data was not showing up in query responses.

    Remediation
    • Rolled back the compaction change
    • Restored recent data first, rebuilt from the Kafka logs
    • Changed the restoration process to make historical data recovery more efficient
    • Restored historical data from the weekly TSM snapshots
    • Kept the cluster online and restored data incrementally so ingest and queries could continue
    Action items
    • Expand test coverage for changes to critical storage components such as the compactor, with scenarios closer to production-scale workloads
    • Strengthen safeguards around compaction so failures are detected and handled safely
    • Review restoration workflows and procedures to reduce elevated TTBR and slow queries during large restores
    • Revisit the backup and restore strategy to speed up cluster-wide restorations
    View case
  2. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021
    Symptoms

    From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

    Service outageHigh latencyScale-up failureCloud API rate limit exhausted
    Trigger

    At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

    Root cause
    Faulty deployment
    Component
    DatabaseNode pool
    Contributing factor
    Ineffective rollbackExcessive blast radiusGap in automated tests
    Detection

    Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

    Remediation
    • Rolled back the field type change, which did not clear the errors
    • Wrote, built and manually deployed a change that made the distribution system ignore the field
    • Manually scaled the primary compute cluster for the expected thundering herd of work
    • Monitored until processing was normal and resolved the incident
    Action items
    • Fix the gaps found in the testing strategy
    • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
    • Make recovery faster when unforeseen failures occur
    View case
Check your cluster for this

These 2 cases involve Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
Database2Database backups and snapshots1Node pool1
Contributing factor
Gap in automated tests2Excessive blast radius1Ineffective rollback1Missed in review1Unrepresentative staging1Unsafe failure behaviour1
Organisation
CircleCI1InfluxData1
  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026
  2. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021