Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Unrepresentative staging

스테이징 불일치 · staging-not-representative

14 cases · 2019 to 2026

Root cause Configuration change4 Breaking change in an upgrade4 Capacity exhaustion3 Software defect2 Cloud provider outage1

Component Ingress controller5 Node pool4 Admission webhook2 Configuration templating2

Symptom Service outage9 High error rate4 Throughput degradation3 DNS resolution failures2

Organisation OpenAI2 UK Ministry of Justice2 DFDS1 Grafana Labs1 Honeycomb

Cases

IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storageLablup Inc · 2026
Root cause Capacity exhaustion
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026
Root cause Breaking change in an upgrade

View on the graph

Kubernetes API server2

Shared network filesystem2

Telemetry agent2

In-memory cache1

Cloud load balancer1

Cluster autoscaler1

Container runtime1

Application database1

Client-side DNS resolver and cache1

GKE control plane1

Horizontal pod autoscaler1

Node launch template1

Container memory limit1

Pod scheduling and placement1

Service discovery1

Training data loader1

Web application firewall rules1

Monitoring outage2

Load imbalance2

Autoscaler misbehaviour1

Scale-up failure1

Control plane lag1

Control plane unresponsive1

Container crash loop1

Database overload1

Deploys failing1

Intermittent connection failures1

Sustained high node CPU1

Pods stuck in ContainerCreating1

Probe failures1

Resources deleted1

Stale endpoint routing1

1

IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count

Hugging Face · 2025

Root cause Capacity exhaustion

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

Root cause Capacity exhaustion

IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025

Root cause Software defect

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024

Root cause Breaking change in an upgrade

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

Root cause Configuration change

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

Root cause Configuration change

IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021

Root cause Breaking change in an upgrade

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage