Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Limited debug access

디버그 접근 제한 · limited-debug-access

13 cases · 2019 to 2025

Root cause Configuration change3 Software defect3 Capacity exhaustion2 Certificate or credential expiry1 Cloud provider outage1 Dependency failure1 Hardware failure1 Breaking change in an upgrade1

Component Kubernetes API server5 etcd2 NCCL2 Node operating system image2 Node pool

Symptom Control plane unresponsive5 Service outage4 GPU unavailable3 High error rate3

Organisation OpenAI3 Anthropic1 BigScience1 DigitalOcean1 Fullstaq

Cases

IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responsesAnthropic · 2025
Root cause Software defect
IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025
Root cause Breaking change in an upgrade
A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

View on the graph

2

Telemetry agent2

Cloud controller manager1

Cluster autoscaler1

Client-side DNS resolver and cache1

GKE control plane1

GPU hardware and memory1

InfiniBand fabric1

Model request routing1

Object storage1

Cluster authentication provider1

Persistent volumes and stateful sets1

RBAC and service accounts1

Slurm job scheduler1

systemd-networkd1

Control plane lag2

Training job failure2

Scale-up failure1

Cloud API rate limit exhausted1

Container crash loop1

Credentials unavailable1

Deploys failing1

DNS resolution failures1

Monitoring outage1

Misrouted requests1

Response quality degradation1

Stale or inconsistent API reads1

1

IL-0085

Magic Pages · 2025

Root cause Capacity exhaustion

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

Root cause Configuration change

IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1

Root cause Software defect

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

Root cause Configuration change

IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its typeHyperconnect · 2023

Root cause Software defect

IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

Root cause Hardware failure

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion