Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

GPU unavailable

GPU 인식 불가 · gpu-unavailable

10 cases · 2021 to 2026

Root cause Dependency failure5 Hardware failure2 Faulty deployment1 Software defect1 Breaking change in an upgrade1

Component GPU driver4 NCCL4 InfiniBand fabric3 Node operating system image3 PyTorch

Contributing factor Unverified assumption3 Limited debug access3 Automation amplification2 Single point of failure2

Organisation BigScience1 Crusoe Cloud1 Google Cloud1 Imbue1 Inflab

Cases

IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026
Root cause Faulty deployment
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1
Root cause Dependency failure
Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False

View on the graph

3

Data center power and cooling2

GPU hardware and memory2

Object storage1

Shared network filesystem1

Slurm job scheduler1

systemd-networkd1

Telemetry agent1

Unreviewed defaults1

Monitoring blind spot1

No staged rollout1

Overlapping failures1

Partial rollout1

Missed in review1

1

IL-0171

Inflab · 2025

Root cause Software defect

IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025

Root cause Breaking change in an upgrade

IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024

Root cause Hardware failure

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

Root cause Dependency failure

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure

IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1Crusoe Cloud · 2023

Root cause Dependency failure

IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

Root cause Hardware failure

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure