Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No headroom

여유 용량 부재 · no-spare-capacity

13 cases · 2019 to 2026

Root cause Capacity exhaustion3 Software defect3 Configuration change2 Cloud provider outage1 Dependency failure1 Faulty deployment1 Operator error1 Breaking change in an upgrade1

Component Container memory limit4 Node pool4 Cluster autoscaler2 Model hub2

Symptom Service outage9 High error rate4 High latency4 OOMKilled4

Organisation OpenAI2 2i2c1 AdGuard VPN1 Buildkite1 Google Cloud1

Cases

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025
Root cause Configuration change
IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

View on the graph

Persistent volumes and stateful sets2

In-memory cache1

Cloud load balancer1

ConfigMaps and Secrets1

Application database1

Client-side DNS resolver and cache1

GKE control plane1

GPU hardware and memory1

Horizontal pod autoscaler1

Model inference server1

InfiniBand fabric1

Ingress controller1

Kubernetes API server1

Model request routing1

Object storage1

Pod priority and preemption1

Probe failures3

DNS resolution failures2

Intermittent timeouts2

Autoscaler misbehaviour1

Scale-up failure1

Control plane lag1

Container crash loop1

Database overload1

GPU unavailable1

Intermittent connection failures1

Monitoring outage1

Training job failure1

Saifeddine Rajhi1

UK Ministry of Justice1

Wealthy · 2025 · sev1

Root cause Breaking change in an upgrade

IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025

Root cause Faulty deployment

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024

Root cause Cloud provider outage

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

Root cause Capacity exhaustion

IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024

Root cause Software defect

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

Root cause Software defect

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change