Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

GPU driver

GPU 드라이버 · gpu-driver

5 cases · 2024 to 2026

Root cause Dependency failure2 Software defect2 Faulty deployment1

Contributing factor Unverified assumption3 Automation amplification2 Monitoring blind spot2 Change with a delayed effect1

Symptom GPU unavailable4 Control plane unresponsive2 Link flapping1 Monitoring outage1

Organisation Barush Mendez1 Google Cloud1 Imbue1 Inflab1 Linode1

Cases, by severity

IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026
Root cause Faulty deployment
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025
Root cause Software defect

View on the graph

Ignored early signal1

Partial rollout1

Missed in review1

Single point of failure1

Node recreation loop1

Service outage1

Throughput degradation1

Training job failure1

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

Barush Mendez · 2025

Root cause Software defect

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

Root cause Dependency failure

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure