Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Overlapping failures

장애 중첩 · overlapping-failures

14 cases · 2017 to 2026

Root cause Cloud provider outage4 Software defect4 Dependency failure2 Capacity exhaustion1 Configuration change1 DNS or network fault1 Operator error1

Component Cluster autoscaler3 Node pool3 Application database2 Data center power and cooling2

Symptom Service outage10 Pods Pending4 High error rate3 High latency3

Organisation Hugging Face2 Anthropic1 CoW Protocol1 Crusoe Cloud1 DFDS

Cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025
Root cause Capacity exhaustion
Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responses

View on the graph

Ingress controller2

Kubernetes API server2

Cloud block storage1

GKE control plane1

Horizontal pod autoscaler1

InfiniBand fabric1

Node launch template1

Container memory limit1

Model request routing1

Node root disk1

Node operating system image1

Persistent volumes and stateful sets1

Container registry and its credentials1

Shared network filesystem1

Training data loader1

Scale-up failure2

Intermittent connection failures2

Intermittent timeouts2

Autoscaler misbehaviour1

Control plane lag1

Control plane unresponsive1

Database overload1

Deploys failing1

GPU unavailable1

Image pull failures1

Monitoring outage1

IP address exhaustion1

Misrouted requests1

Response quality degradation1

Stale endpoint routing1

Throughput degradation1

1

IL-0022

Anthropic · 2025

Root cause Software defect

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024

Root cause Cloud provider outage

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024

Root cause Configuration change

IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1Crusoe Cloud · 2023

Root cause Dependency failure

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020

Root cause DNS or network fault

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect