Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Root cause

Dependency failure

의존성 장애 · dependency-failure

13 cases · 2019 to 2026

Component InfiniBand fabric5 Data center power and cooling3 NCCL3 Node operating system image3 GPU driver2 Node pool2 PyTorch2 Admission webhook1 Cloud block storage1 Container runtime1 Credential broker for pods1 Application database1 Authoritative DNS servers and records1 etcd1 GPU hardware and memory1 Liveness and readiness probes1 Istio1 Object storage1 Container registry and its credentials1 Service discovery1 Shared network filesystem1 Telemetry agent1 VPC DNS resolver1

Contributing factor Monitoring blind spot4 Automation amplification3 Unreviewed defaults3 Single point of failure3

Symptom GPU unavailable5 Service outage3 Training job failure3 Control plane unresponsive2

Organisation Linode2 Crusoe Cloud1 Databricks1 FREE NOW1 Google Cloud

Cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1
IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loopYonghyun Kim · 2025
IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hangLinode · 2025

View on the graph

Excessive blast radius2

Overlapping failures2

Unverified assumption1

Unpatched known bug1

Limited debug access1

No alert on limit1

Partial rollout1

Container crash loop2

Probe failures2

Scale-up failure1

Credentials unavailable1

DNS resolution failures1

Image pull failures1

Monitoring outage1

Storage disconnected or stale mounts1

Throughput degradation1

Load imbalance1

1

Tetrate (unnamed U.S. federal customer)1

IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

Tetrate (unnamed U.S. federal customer) · 2025 · sev1

IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clustersLinode · 2025

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1Crusoe Cloud · 2023

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1