Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Throughput degradation

처리량 저하 · throughput-degraded

7 cases · 2020 to 2026

Root cause Capacity exhaustion2 Configuration change1 Dependency failure1 DNS or network fault1 Software defect1 Breaking change in an upgrade1

Component NCCL2 Shared network filesystem2 Admission webhook1 Cilium1

Contributing factor Monitoring blind spot4 Unrepresentative staging3 Unverified assumption2 Unreviewed defaults2

Organisation Airbnb1 ByteDance1 Datadog1 Hugging Face1 Imbue1

Cases

IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storageLablup Inc · 2026
Root cause Capacity exhaustion
IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025
Root cause Capacity exhaustion

View on the graph

Horizontal pod autoscaler1

InfiniBand fabric1

NIC, cable and switch port link1

Node operating system image1

systemd-networkd1

Training data loader1

Change with a delayed effect2

Automation amplification1

Ignored early signal1

No staged rollout1

Overlapping failures1

Partial rollout1

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

Imbue · 2024

Root cause Dependency failure

IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024

Root cause DNS or network fault

IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

Root cause Breaking change in an upgrade

IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

Root cause Configuration change

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect