IL-0264 Healthchecks.io · 2025-11-01networking

Healthchecks.io requests slowed and timed out as HAProxy load balancers ran out of connection slots during a ping flood

Healthchecks.io rate-limits in two layers to keep ping floods from overloading its database: HAProxy on the load balancer servers applies a lax first limit and passes good requests to NGINX on the application servers, which applies stricter rate limiting and geo-blocking. For clients over the limit HAProxy can deny (return HTTP 429), tarpit (wait, then return HTTP 429, holding the socket during the wait), silently drop or reject the connection. Healthchecks.io used tarpit with a 1-second wait for the spammiest clients. Tarpit slows clients that send requests one after another, but can overwhelm the load balancer when a client sends a very large number of requests in parallel.

When a misconfigured client sent a sustained 300 requests per second for one UUID, the tarpit did not slow it down, and the HAProxy servers hit their connection slot limits and rejected legitimate requests. One load balancer was removed from DNS rotation at 23:39, shifting its load to the others. After being woken at 00:11, the operator identified the client's addresses, a single IPv4 address and a /64 IPv6 subnet, and at 01:06 changed the load balancer configuration to reject TCP requests from them, after which things went back to normal. At 08:40 the rate limiting was changed to return HTTP 429 immediately instead of using tarpit, and in the following days the operator added an HAProxy request timeout, raised maxconn, extended rate limiting to hchk.io, added a Netdata alert on HAProxy frontend sessions and changed the HAProxy log format.

Severity: –
Scope: healthchecks.io website and hc-ping.com ping endpoints: some requests took several seconds or timed out; false monitoring alerts for users
Users affected: –
Region: –
Time to detect: 41m
Time to resolve: 1h 40m

Symptoms

Between 23:30 UTC on 1 November and 1:10 UTC on 2 November 2025, both the Healthchecks.io website (healthchecks.io) and the ping endpoints (hc-ping.com) had intermittent connectivity problems: some requests took several seconds to finish and others timed out entirely. The load balancers were running out of connection slots and rejecting legitimate requests, and Healthchecks.io users received false monitoring alerts.

At 23:24 UTC ping requests for one specific UUID rose from 10 requests per second to a sustained 300 requests per second from a misconfigured client. At 23:39 one of the load balancers began rejecting connections and was automatically removed from DNS rotation, which put more load on the remaining load balancers and started a chain reaction. The operator was woken by notifications at 00:11.

HAProxy on the load balancers rate-limited particularly spammy clients with the "tarpit" action (http-request tarpit), which waits before returning HTTP 429 and keeps a network socket tied up while it waits. The wait was set to 1 second to limit the risk of running out of connection slots. The flooding client did not slow down, and at its request rate the waiting requests used up the HAProxy connection slots. The post-mortem shows a Netdata graph of the aggregate number of open IPv4 and IPv6 sockets on the load balancer servers.

Symptom Intermittent timeouts High latency Intermittent connection failures

Root cause

Root cause Capacity exhaustion

Component HAProxy

Contributing factor Automation amplification No alert on limit

Trigger: A misconfigured client flooded the ping endpoint, raising requests for a single UUID from 10 to a sustained 300 requests per second.

Detection

Monitoring notifications woke the operator at 00:11 UTC.

Remediation

Identified the offending client addresses, a single IPv4 address and a /64 IPv6 subnet
Changed the load balancer configuration to reject TCP requests from those addresses

Action items

Rate-limit with HTTP 429 returned immediately (the "Deny" method) instead of the "tarpit" method
Add the missing timeout http-request 5s to the HAProxy configuration to guard against Slowloris attacks
Increase the maxconn parameter in the HAProxy configuration
Apply the load balancer rate-limiting rules to the hchk.io endpoints as well as hc-ping.com
Add a Netdata alert when active HAProxy frontend sessions pass thresholds well below maxconn
Change the HAProxy logging format to make the logs easier to use

Lessons

The tarpit rate-limiting method only slows clients that make requests one after another; a client sending many requests in parallel is not slowed, and the sockets held during the wait can exhaust the load balancer's connection slots. The operator called it one more lesson learned and switched rate-limited requests to the "Deny" method.

Sources

https://blog.healthchecks.io/2025/11/post-mortem-outage-on-november-1/ · blog · archived copy

Related cases

IL-0578 Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of secondsGMO Pepabo · 2020
same root cause · shared HAProxy
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020
same root cause · shared HAProxy

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, HAProxy among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0264
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0264/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.