Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No alert on limit

한계 경보 부재 · no-alert-on-limit

12 cases · 2019 to 2026

Root cause Capacity exhaustion5 Configuration change2 Software defect2 Certificate or credential expiry1 Dependency failure1 Faulty deployment1

Component Node pool5 Container memory limit3 etcd2 Kubernetes API server2

Symptom Monitoring outage5 High error rate4 Control plane unresponsive3 Service outage3

Organisation UK Ministry of Justice5 dbhang1 loveholidays1 Marcin Cuber1

Cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

View on the graph

Telemetry agent2

Cloud block storage1

In-memory cache1

Cluster autoscaler1

conntrack table1

Application database1

Data center power and cooling1

Elasticsearch or OpenSearch1

InfiniBand fabric1

Node launch template1

Model request routing1

Node root disk1

Pod scheduling and placement1

Connection resets1

Control plane lag1

Container crash loop1

Database overload1

Deploys failing1

Image pull failures1

Intermittent connection failures1

Intermittent timeouts1

IP address exhaustion1

Node disk full1

Sustained high node CPU1

Pods stuck in ContainerCreating1

Probe failures1

IL-0020

UK Ministry of Justice · 2025

Root cause Faulty deployment

IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1

Root cause Software defect

IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

Root cause Software defect

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

Root cause Configuration change

IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expireddbhang · 2021

Root cause Certificate or credential expiry

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change