Incident Library

IL-0379 InfluxData · 2026-09-27databaseazure

InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

In InfluxDB Cloud, all data in a cluster is stored on redundant storage pods spread over many physical nodes. The pods read time-series data through memory-mapped files, and the host operating system keeps the most used parts of those files in a page cache made of the memory the pods do not use. In Azure West Europe each pod held about 365 GB of data, and the cache on a host could not hold the data its pods were querying. Reads that would normally come from the cache became page faults served from disk. Each pod read about 90 MB/s, so a host with ten pods read roughly 990 MB/s, close to the throughput ceiling of each host.

At that ceiling, index lookups on the write path waited on disk reads, writes fell behind and pods failed their replication-lag health check. Slow disk also made health and readiness checks time out, so healthy pods were removed and restarted, and each 30 to 85 minute recovery left the partner pod alone under load and added rebuild reads. From 27 to 30 September InfluxData increased query-layer capacity, restarted pods, raised the readiness probe timeout, doubled recovery concurrency and raised a cleanup timeout to 24 hours, which stopped the forced restarts from a lost file-reference cleanup, yet the degradation returned at each morning peak. InfluxData believes this cache shortage was also the root cause of the earlier incidents on this cluster on 7, 11, 15 and 16 September.

On 30 September at 11:59 UTC analysis showed that the cache per pod was too small and that disk reads were at the host ceiling. From 12:43 UTC InfluxData raised the resource request of each storage pod so that each host held fewer pods, which gave each pod twice the memory for page cache and twice the disk throughput. Within an hour query abandonment fell from 31% to 0.1% and page faults and disk reads dropped sharply. The morning peak on 1 October had 0.06% abandonment, and the incident was declared resolved internally at 09:48 UTC.

Severity
–
Scope
Queries in InfluxDB Cloud Serverless on Azure West Europe were slow and timed out, with about 55% abandoned at the worst peak; writes were accepted, but new data on some partitions was not immediately queryable
Users affected
Customers in the Azure West Europe region
Region
Azure West Europe
Time to detect
–
Time to resolve
–

Symptoms

From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

The degradation came back at each morning peak. On 28 September storage pods fell behind on writes and failed health checks; on 29 September throughput seemed to recover and then 36% of queries were abandoned; on 30 September about 55% were abandoned. Pods in the region spent 8.9% of the day past the replication-lag threshold, against 0.09% in a comparable region, and at peak both replicas of the busiest partitions fell behind at the same time.

Memory looked normal on every metric InfluxData monitored, because the storage process uses little memory and the host page cache takes the rest, so a cache shortage did not show as memory pressure. Slow disk made health and readiness checks time out, so healthy pods were taken out of rotation and restarted; each restart took 30 to 85 minutes to recover, while the pod's partner carried the load alone and the rebuild added more disk load. A file-reference safeguard also forced more than 50 pod restarts a day, where the normal number is 0.

Doubling query-layer capacity, restarting pods, raising readiness and cleanup timeouts and doubling recovery concurrency cleared many warning signs but did not remove the cause. A controller lock, a per-disk limit and the load of a single tenant were examined and ruled out. Earlier incidents on the same cluster on 7, 11, 15 and 16 September had shown the same symptom with different triggers.

Symptom High latencyIntermittent timeoutsReplication lagProbe failures

Root cause

Root cause Capacity exhaustion

Component DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement
Contributing factor Monitoring blind spotAutomation amplification
Trigger
–

Detection

A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

Remediation

Action items

Lessons

Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication, Liveness and readiness probes among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.