Incident Library

Organisation

Hugging Face

Hugging Face · hugging-face

2 cases · 2024 to 2025

Root cause Capacity exhaustion1Software defect1
Component Cluster autoscaler1Application database1Model hub1Shared network filesystem1Training data loader1
Contributing factor Overlapping failures2Unverified assumption1No headroom1Retry storm1Unrepresentative staging1
Symptom Database overload1High latency1Service outage1Throughput degradation1

Cases

  1. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025

    Root cause Capacity exhaustion

  2. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

    Root cause Software defect