IL-0024 Hugging Face · 2024-04-22ai-infra
From 08:45 CET on 22 April 2024 until 10:03 on 24 April the Hugging Face Hub had an extended outage with three separate crashes, during which the website was slow or inaccessible for most users. The write-up names Gitaly, AWS WAF, MongoDB and MongoDB Atlas as parts of the Hub's production architecture, and the Hub itself runs on a Kubernetes cluster.
The cause was a set of very resource-intensive API calls that read from the *_info_cache collections in MongoDB; the call behind the 23 April crash used space_info_cache, and datasets_info_cache is one of the three collections that put a significant strain on the database. Requests to the lighteval/mmlu dataset revision endpoint rose sharply right before the first crash. Requests to the sentence-transformers/all-mpnet-base-v2 model revision endpoint, which lists 300+ associated spaces, lined up with the crash on 23 April, and traffic to that endpoint correlated with database network out, which had come before several of the crashes. A k6 script set up to send that one request from 512 virtual users over ten minutes made the Hub slow or unresponsive when the team ran it on 24 April.
A slow database turned into a collapse because the Hub's Node.js code did not cancel database requests when users cancelled theirs. When a page was slow and users refreshed it, every earlier database request kept running to completion with nobody waiting for the answer, a thundering herd that snowballed once something hit the database hard.
During the outage the team blocked the Java user agent at the firewall, disabled the quicksearch feature, scaled the MongoDB cluster by hand from tier 2 to tier 3, put the Hub into maintenance mode with a 503 page through the WAF and scaled the Hub's pods down to 0, and on 24 April set an aggressive rate limit. Afterwards it added timeouts to requests based on their usual completion time, reworked the queries that were slow or returned large bodies, removed old unused indexes, and planned to move the *_info_cache collections to Redis and make database requests abort when the client aborts.
At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.
In the MongoDB Atlas metrics two read-only nodes were down and kept restarting. The database cluster autoscaled once, from tier 1 to tier 2, and was HEALTHY again at 07:33 UTC, but the Hub stayed unstable even though the database looked healthy. A spike of traffic from an internal source with a Java/17 user agent was suspected and blocked at the firewall at 10:22; that reduced the load but did not fix the problem. The nodes of the Hub's Kubernetes cluster had also not scaled automatically, because of a configuration issue.
The outage came back twice. At 21:30 on 23 April the number of database connections and the system memory rose sharply and the MongoDB primary ran out of memory; the cluster elected a new primary, the same thing happened, and it elected the previous primary again. At 08:00 on 24 April the Hub was slow and unresponsive with the same pattern, and at 10:03 suspicious metrics pointed to another crash.
The database metrics showed the *_info_cache collections were overused, with poor average latency and very high network out. Going through all the existing charts and graphs showed nothing else, and identifying the root cause took two to three days; it came only after the team built a new graph that broke requests down by URL.
Symptom Service outageHigh latency
Root cause Software defect
Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.
Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Application database
IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clustersLinode · 2025same root cause · shared Cluster autoscaler
IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Application database
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.