Cases2 cases

1 to 2 of 2

1. IL-0264 Healthchecks.io requests slowed and timed out as HAProxy load balancers ran out of connection slots during a ping flood

   Healthchecks.io2025

   Symptoms

   Between 23:30 UTC on 1 November and 1:10 UTC on 2 November 2025, both the Healthchecks.io website (healthchecks.io) and the ping endpoints (hc-ping.com) had intermittent connectivity problems: some requests took several seconds to finish and others timed out entirely. The load balancers were running out of connection slots and rejecting legitimate requests, and Healthchecks.io users received false monitoring alerts.

   Intermittent timeoutsHigh latencyIntermittent connection failures

   Trigger

   A misconfigured client flooded the ping endpoint, raising requests for a single UUID from 10 to a sustained 300 requests per second.

   Root cause

   Capacity exhaustion

   Component

   HAProxy

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   Monitoring notifications woke the operator at 00:11 UTC.

   Remediation

   • Identified the offending client addresses, a single IPv4 address and a /64 IPv6 subnet
   • Changed the load balancer configuration to reject TCP requests from those addresses

   Action items

   • Rate-limit with HTTP 429 returned immediately (the "Deny" method) instead of the "tarpit" method
   • Add the missing timeout http-request 5s to the HAProxy configuration to guard against Slowloris attacks
   • Increase the maxconn parameter in the HAProxy configuration
   • Apply the load balancer rate-limiting rules to the hchk.io endpoints as well as hc-ping.com
   • Add a Netdata alert when active HAProxy frontend sessions pass thresholds well below maxconn
   • Change the HAProxy logging format to make the logs easier to use

   Lessons

   The tarpit rate-limiting method only slows clients that make requests one after another; a client sending many requests in parallel is not slowed, and the sockets held during the wait can exhaust the load balancer's connection slots. The operator called it one more lesson learned and switched rate-limited requests to the "Deny" method.

   View case
2. IL-0578 Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of seconds

   GMO Pepabo2020

   Symptoms

   In a GMO Pepabo production setup, delivery of static content was delayed by tens of seconds. The static content was served by nginx on upstream hosts behind HAProxy, which ran as the HTTP load balancer.

   High latency

   Root cause

   Capacity exhaustion

   Component

   HAProxyLinux kernelCloud load balancer

   Contributing factor

   Unreviewed defaults

   Remediation

   • Tuned net.ipv4.tcp_mem with sysctl on the HAProxy host, which ended the delays immediately

   Lessons

   On a VM with little RAM, the boot-time default of net.ipv4.tcp_mem can be low enough to push TCP into memory pressure. TCP out of memory is not the OOM killer, and net.ipv4.tcp_mem is a system-wide limit counted in pages, unlike the per-socket net.ipv4.tcp_rmem and net.ipv4.tcp_wmem counted in bytes. The values to watch are TCP mem in /proc/net/sockstat, press in /proc/net/protocols, and TCPMemoryPressures, TCPMemoryPressuresChrono and TCPAbortOnMemory in /proc/net/netstat.

   View case

By root cause1