Cases2 cases

1 to 2 of 2

1. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times

   Hugging Face2024

   Symptoms

   At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.

   Service outageHigh latencyDatabase overload

   Trigger

   Bursts of requests to revision endpoints that read heavily from the *_info_cache collections: a call to https://huggingface.co/api/datasets/lighteval/mmlu/revision/main rose sharply right before the first crash, and a call to https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2/revision/main, a model with 300+ associated spaces, lined up with the crash at 21:30 on 23 April.

   Root cause

   Software defect

   Component

   Application databaseModel hubCluster autoscaler

   Contributing factor

   No headroomRetry stormOverlapping failures

   Detection

   Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.

   Remediation

   • Added a firewall rule blocking all requests with a Java/* user agent
   • Disabled the quicksearch feature on the Hub
   • Scaled the MongoDB Atlas cluster by hand from tier 2 to tier 3
   • Switched the Hub to maintenance mode with a 503 page through AWS WAF and blocked incoming traffic
   • Scaled all Hub pods down to 0 and brought them back
   • Set an aggressive rate limit
   • Added timeouts to requests and reworked database queries that were slow or returned large bodies

   Action items

   • Add timeouts to requests based on their usual completion time
   • Optimise requests that took too long or returned a large body, and rework poorly designed database queries
   • Remove old unused indexes from the database
   • Build new dashboards and monitoring metrics
   • Move the *_info_cache collections to Redis
   • Make it possible to abort any database request when the client aborts theirs

   Lessons

   Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.

   View case
2. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA

   loveholidays2019

   Symptoms

   On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

   Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour

   Trigger

   A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

   Root cause

   Cloud provider outage

   Component

   GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler

   Contributing factor

   Unrepresentative stagingOverlapping failuresIgnored early signal

   Detection

   The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

   Remediation

   • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
   • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
   • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
   • used kubectl port-forward in place of the broken internal Ingresses
   • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back

   Action items

   • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
   • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
   • Re-evaluate termination grace periods that were set to arbitrary large values
   • Read the GKE release notes before every cluster upgrade
   • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
   • Keep environments as identical as possible, including resource names

   Lessons

   Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

   View case

By root cause2