Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Monitoring blind spot

모니터링 사각지대 · monitoring-blind-spot

43 cases · 2017 to 2026

Root cause Capacity exhaustion13 Configuration change10 Software defect8 Dependency failure4 Faulty deployment3 Breaking change in an upgrade2 Certificate or credential expiry1 Cloud provider outage1 unknown1

Component Kubernetes API server9 kubelet8 etcd6 Admission webhook5

Symptom Service outage17 High error rate13 High latency8 Monitoring outage8

Organisation DFDS5 OpenAI3 2i2c1 Airbnb1 Anthropic1

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026
Root cause Capacity exhaustion

View on the graph

Ingress controller5

Telemetry agent5

Cloud load balancer4

Container memory limit4

Container CPU limit3

Horizontal pod autoscaler3

InfiniBand fabric3

Container runtime2

Application database2

Client-side DNS resolver and cache2

Node root disk2

Node operating system image2

Persistent volumes and stateful sets2

Cloud block storage1

In-memory cache1

Automated capacity manager1

Cloud controller manager1

Cluster autoscaler1

Cluster dashboard1

Configuration templating1

conntrack table1

Data center power and cooling1

Elasticsearch or OpenSearch1

GKE control plane1

Language runtime settings1

Node launch template1

Model request routing1

Pod termination1

RBAC and service accounts1

Container registry and its credentials1

Shared network filesystem1

Control plane unresponsive6

DNS resolution failures5

Deploys failing4

Intermittent timeouts4

Throughput degradation4

Control plane lag3

CPU throttling3

Intermittent connection failures3

Probe failures3

Resources deleted3

Scale-up failure2

Container crash loop2

Stale endpoint routing2

Training job failure2

Unauthorised access or a foreign workload2

Autoscaler misbehaviour1

Cloud API rate limit exhausted1

GPU unavailable1

Image pull failures1

Node disk full1

Sustained high node CPU1

Node recreation loop1

Outbound connection exhaustion1

Pods stuck in ContainerCreating1

Misrouted requests1

Response quality degradation1

Civis Analytics1

Microsoft Azure1

Onkar Raghunath Shelke1

PrometheusKube1

UK Ministry of Justice1

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

Buildkite · 2026

Root cause Capacity exhaustion

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026

Root cause Dependency failure

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026

Root cause Dependency failure

IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storageLablup Inc · 2026

Root cause Capacity exhaustion

IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loadsGitHub · 2026

Root cause Configuration change

IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error ratesOpenAI · 2026

Root cause Software defect

IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026

Root cause Capacity exhaustion

IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthyYonghyun Kim · 2025

Root cause Faulty deployment

IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responsesAnthropic · 2025

Root cause Software defect

IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025

Root cause Capacity exhaustion

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

Root cause Capacity exhaustion

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025

Root cause Software defect

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

Root cause Dependency failure

IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024

Root cause Breaking change in an upgrade

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023

Root cause Capacity exhaustion

IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023

Root cause Configuration change

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

Root cause Configuration change

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutesDFDS · 2022

Root cause Certificate or credential expiry

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

Root cause Configuration change

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

Root cause Faulty deployment

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect

IL-0041 fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020

Root cause Software defect

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

Root cause Dependency failure

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

Root cause Configuration change

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion