Incident Library
Applied filters ComponentPod scheduling and placementSymptomIntermittent timeoutsFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting
    InfluxData2026
    Symptoms

    From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

    High latencyIntermittent timeoutsReplication lagProbe failures
    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement
    Contributing factor
    Monitoring blind spotAutomation amplification
    Detection

    A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

    Remediation
    • Doubled query-layer capacity
    • Restarted stuck storage pods one at a time
    • Raised the readiness probe timeout and rolled it out fleet-wide
    • Doubled recovery concurrency after a restart
    • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
    • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod
    Action items
    • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
    • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
    • Keep tracking down the storage-engine reference leak
    • Move the disk check out of the health path
    Lessons

    Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

    View case
Check your cluster for this

This case involves Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement
Contributing factor
Automation amplificationMonitoring blind spot
Organisation
InfluxData
  1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting
    InfluxData2026