Cases2 cases

1 to 2 of 2

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0264 Healthchecks.io requests slowed and timed out as HAProxy load balancers ran out of connection slots during a ping flood

   Healthchecks.io2025

   Symptoms

   Between 23:30 UTC on 1 November and 1:10 UTC on 2 November 2025, both the Healthchecks.io website (healthchecks.io) and the ping endpoints (hc-ping.com) had intermittent connectivity problems: some requests took several seconds to finish and others timed out entirely. The load balancers were running out of connection slots and rejecting legitimate requests, and Healthchecks.io users received false monitoring alerts.

   Intermittent timeoutsHigh latencyIntermittent connection failures

   Trigger

   A misconfigured client flooded the ping endpoint, raising requests for a single UUID from 10 to a sustained 300 requests per second.

   Root cause

   Capacity exhaustion

   Component

   HAProxy

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   Monitoring notifications woke the operator at 00:11 UTC.

   Remediation

   • Identified the offending client addresses, a single IPv4 address and a /64 IPv6 subnet
   • Changed the load balancer configuration to reject TCP requests from those addresses

   Action items

   • Rate-limit with HTTP 429 returned immediately (the "Deny" method) instead of the "tarpit" method
   • Add the missing timeout http-request 5s to the HAProxy configuration to guard against Slowloris attacks
   • Increase the maxconn parameter in the HAProxy configuration
   • Apply the load balancer rate-limiting rules to the hchk.io endpoints as well as hc-ping.com
   • Add a Netdata alert when active HAProxy frontend sessions pass thresholds well below maxconn
   • Change the HAProxy logging format to make the logs easier to use

   Lessons

   The tarpit rate-limiting method only slows clients that make requests one after another; a client sending many requests in parallel is not slowed, and the sockets held during the wait can exhaust the load balancer's connection slots. The operator called it one more lesson learned and switched rate-limited requests to the "Deny" method.

   View case

By root cause1