Incident Library
Applied filters Root causeSoftware defectComponentApplication databaseSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
    Symptoms

    At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.

    Service outageHigh latencyDatabase overload
    Trigger

    Bursts of requests to revision endpoints that read heavily from the *_info_cache collections: a call to https://huggingface.co/api/datasets/lighteval/mmlu/revision/main rose sharply right before the first crash, and a call to https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2/revision/main, a model with 300+ associated spaces, lined up with the crash at 21:30 on 23 April.

    Root cause
    Software defect
    Component
    Application databaseModel hubCluster autoscaler
    Contributing factor
    No headroomRetry stormOverlapping failures
    Detection

    Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.

    Remediation
    • Added a firewall rule blocking all requests with a Java/* user agent
    • Disabled the quicksearch feature on the Hub
    • Scaled the MongoDB Atlas cluster by hand from tier 2 to tier 3
    • Switched the Hub to maintenance mode with a 503 page through AWS WAF and blocked incoming traffic
    • Scaled all Hub pods down to 0 and brought them back
    • Set an aggressive rate limit
    • Added timeouts to requests and reworked database queries that were slow or returned large bodies
    Action items
    • Add timeouts to requests based on their usual completion time
    • Optimise requests that took too long or returned a large body, and rework poorly designed database queries
    • Remove old unused indexes from the database
    • Build new dashboards and monitoring metrics
    • Move the *_info_cache collections to Redis
    • Make it possible to abort any database request when the client aborts theirs
    Lessons

    Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.

    View case
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
    Symptoms

    Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

    Data lossService outage
    Trigger

    A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

    Root cause
    Software defect
    Component
    Application databasePersistent volumes and stateful setsContainer memory limit
    Contributing factor
    No staged rolloutSingle point of failureUnverified assumption
    Remediation
    • restored the Redis data from the backup prepared before the rollout
    Action items
    • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
    • Reconsider the use of the Spotahome Redis Operator
    • Clean up the Redis databases and fix the application code that created the excess keys
    • Consider Redis built-in clustering and sharding
    Lessons

    Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

    View case
Check your cluster for this

These 2 cases involve Application database and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Application database2Cluster autoscaler1Container memory limit1Model hub1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption1No headroom1No staged rollout1Overlapping failures1Retry storm1Single point of failure1
Organisation
Hugging Face1Palark1
  1. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020