Incident Library
Applied filters Root causeSoftware defectOrganisationHubSpotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024
    Symptoms

    Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

    Probe failuresHigh latencyHigh error rate
    Trigger

    Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

    Root cause
    Software defect
    Component
    Ingress controllerLiveness and readiness probes
    Contributing factor
    Automation amplificationNo headroom
    Detection

    Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

    Remediation
    • Tried to increase the capacity of the remaining load balancer instances
    • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal
    Action items
    • Change the health check configuration so health checks do not time out falsely
    • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
    • Increase the capacity of the load balancers for sudden traffic spikes
    • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
    • Update the status page automatically for large-scale outages
    • Integrate the monitoring systems more tightly with the communication tools
    • Expand load testing to simulate high-traffic scenarios
    • Introduce chaos engineering to test platform resilience regularly
    • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
    • Add auto-scaling that adjusts resource capacity to traffic patterns
    Lessons

    HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

    View case
Check your cluster for this

This case involves Liveness and readiness probes and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Liveness and readiness probesIngress controller
Contributing factor
Automation amplificationNo headroom
Organisation
HubSpot
  1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024