Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Model hub

모델 허브 · model-hub

2 cases · 2022 to 2024

Root cause Operator error1 Software defect1

Contributing factor No headroom2 Overlapping failures2 No safeguard on destructive change1 Retry storm1 Deferred upgrades1

Symptom Service outage2 Database overload1 High error rate1 High latency1

Organisation Hugging Face1 OpenAI1

Cases, by severity

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024
Root cause Software defect
IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022
Root cause Operator error

View on the graph

Intermittent timeouts1