Cases2 cases

1 to 2 of 2

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale

   MUSINSA2025

   Symptoms

   Shortly after 29CM's Igu Week sale began on 3 November 2025, the servers for the display domain, which serve search results and product listings, showed anomalies. Some pods went down and stopped taking traffic, and the remaining pods received more traffic than they could handle and saturated their Netty event loops.

   Probe failuresIntermittent connection failuresHigh latency

   Trigger

   Igu Week, 29CM's largest Black Friday sale and its busiest traffic of the year, raised Redis network traffic to about four times normal.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheLiveness and readiness probes

   Contributing factor

   Unknown limitMonitoring blind spotHealth check tied to a dependency

   Detection

   Anomalies on the display servers, pods going down and Netty event loops saturating, were the first sign. The Redis cause was found from Redis health check failure logs and then the Redis network metrics.

   Remediation

   • Scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, about twice the baseline network bandwidth

   Action items

   • Extended the monitoring dashboard to show in real time whether network in and out bandwidth exceeds the baseline, with metrics such as aws.elasticache.network_bandwidth_out_allowance_exceeded and aws.elasticache.traffic_management_active
   • Added Datadog alerts that fire when network bandwidth crosses a threshold
   • Added a Caffeine local cache in front of Redis for data that changes rarely
   • Look into compressing cached data with Snappy to reduce Redis network bandwidth
   • Look into storing cached data as Protocol Buffers instead of JSON; measured on the cached Item Document data, Snappy and protobuf together are expected to cut the size by about 73%
   • Keep improving the architecture for traffic growth, building on the cache restructuring, stronger monitoring and read separation already applied

   Lessons

   CPU and memory metrics alone did not show the problem, so Redis needs network monitoring against the node's bandwidth baseline. Burst credits kept the nodes working for about two hours after traffic passed the baseline, which delayed the failure and made its cause slower to find.

   View case

By root cause1