IL-0495 HubSpot · 2024-08-27kubernetesnetworking
HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
HubSpot's platform sends customer traffic through several layers of security and routing systems. One of these layers is a set of internal load balancers deployed on Kubernetes that distribute traffic across the backend services. On 27 August 2024 a routine operation of the traffic management system ran into an issue in the load balancer stack that HubSpot had not known about, although it had been documented outside HubSpot: path handling updates temporarily blocked the responses to health checks. The health checks, which decide whether a load balancer instance is working and ready for traffic, did not complete in time, and the system read those instances as unhealthy.
Several load balancer instances were falsely marked as failed and shut down at the same time. The remaining instances took a sudden surge of connections, 11x the normal peak, and went into congestive collapse. Built-in protections rejected excess connections, but performance for the affected customers was heavily degraded. At 14:59 UTC the team tried to increase the capacity of the remaining instances, at 15:04 UTC new load balancer instances came online and took the pressure off, and by 15:16 UTC the system had fully recovered and traffic was back to normal levels.
- Severity
- –
- Scope
- Logging in, reaching content and using HubSpot tools
- Users affected
- some HubSpot customers
- Region
- –
- Time to detect
- –
- Time to resolve
- 56m
Symptoms
Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.
The first sign came at 14:08 UTC, when automated health checks on the load balancer instances started to slow down during routine operation. At 14:18 UTC the automatic detection of impaired instances marked a subset of the load balancer instances down because their health checks were failing, and began to take them out of traffic. Those instances were not broken: path handling updates in the load balancer stack were blocking the health check responses, so the checks could not finish in time.
At 14:19 UTC the remaining load balancer instances received a massive surge in connections, 11x the normal peak, going from hundreds of thousands to several million connections. At 14:23 UTC automated systems applied connection limits to reduce memory overload, which brought longer response times and failed requests, and monitoring detected widespread performance degradation across the platform. The public status page was updated only at 14:44 UTC.
Root cause
Root cause Software defect
- Trigger
- Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.
Detection
Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.
Remediation
- Tried to increase the capacity of the remaining load balancer instances
- Brought new load balancer instances online, which relieved the pressure and let traffic return to normal
Action items
- Change the health check configuration so health checks do not time out falsely
- Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
- Increase the capacity of the load balancers for sudden traffic spikes
- Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
- Update the status page automatically for large-scale outages
- Integrate the monitoring systems more tightly with the communication tools
- Expand load testing to simulate high-traffic scenarios
- Introduce chaos engineering to test platform resilience regularly
- Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
- Add auto-scaling that adjusts resource capacity to traffic patterns
Lessons
HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.
Sources
Related cases
IL-0053Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025same root cause · shared Ingress controller
IL-0081API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025same root cause · shared Ingress controller
IL-0106A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021same root cause · shared Ingress controller
IL-0062Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019same root cause · shared Ingress controller
IL-0011HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019same root cause · shared Ingress controller
IL-0072High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Ingress controller
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0495
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0495/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.