Cases2 cases

1 to 2 of 2

1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded

   OpenAI2022

   Symptoms

   At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

   Service outageHigh error rateIntermittent timeouts

   Trigger

   A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

   Root cause

   Operator error

   Component

   Ingress controllerModel hub

   Contributing factor

   No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures

   Remediation

   • re-allocated the models from the lost cluster to the remaining two clusters
   • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
   • built workarounds with Azure support after losing the connection to Azure Private Link
   • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
   • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT

   Action items

   • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
   • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
   • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
   • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
   • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
   • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
   • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
   • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
   • Change how model weights are stored so they are more efficient and faster to load

   Lessons

   Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

   View case

By root cause2