Cases3 cases

1 to 3 of 3

1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause1