Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Unverified assumption

미검증 가정 · assumption-unverified

35 cases · 2017 to 2026

Root cause Configuration change9 Software defect8 Capacity exhaustion6 Faulty deployment3 Cloud provider outage2 DNS or network fault2 Operator error2 Dependency failure1 unknown1 Breaking change in an upgrade1

Component Telemetry agent6 Application database5 Ingress controller5 Cloud load balancer4

Symptom Service outage15 High error rate8 High latency8 Intermittent connection failures6

Organisation 2i2c2 DFDS2 Grafana Labs2 Palark2 Adevinta1

Cases

IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
Root cause Configuration change
IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSyncArgo CD users · 2026
Root cause Software defect
A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created

View on the graph

Persistent volumes and stateful sets4

Container memory limit3

Cluster autoscaler2

Configuration templating2

Container runtime2

Client-side DNS resolver and cache2

Kubernetes API server2

Node operating system image2

Pod scheduling and placement2

Pod termination2

Admission webhook1

In-memory cache1

ConfigMaps and Secrets1

conntrack table1

Container CPU limit1

Elasticsearch or OpenSearch1

GKE control plane1

Horizontal pod autoscaler1

InfiniBand fabric1

Jobs and CronJobs1

Application logging library1

Multus meta-CNI1

Node root disk1

Service discovery1

Shared network filesystem1

VPC subnet or pod address range1

Training data loader1

DNS resolution failures3

GPU unavailable3

Sustained high node CPU3

Probe failures3

Resources deleted3

Control plane lag2

Control plane unresponsive2

Node recreation loop2

Throughput degradation2

Autoscaler misbehaviour1

Scale-up failure1

Connection pool exhaustion1

Connection resets1

CPU throttling1

Database overload1

Deploys failing1

File descriptor exhaustion1

Intermittent timeouts1

IP address exhaustion1

Stale endpoint routing1

Training job failure1

Load imbalance1

Civis Analytics1

Toyota Connected1

IL-0097

2i2c · 2026

Root cause Breaking change in an upgrade

IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026

Root cause Faulty deployment

IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025

Root cause Configuration change

IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthyYonghyun Kim · 2025

Root cause Faulty deployment

IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025

Root cause Capacity exhaustion

IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025

Root cause Software defect

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025

Root cause Capacity exhaustion

IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

Root cause Configuration change

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

Root cause Dependency failure

IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

Root cause DNS or network fault

IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every nodeFyra Labs · 2023

Root cause unknown

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutesGitLab · 2021

Root cause Configuration change

IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

Root cause Configuration change

IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0Palark · 2021

Root cause Operator error

IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

Root cause Cloud provider outage

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change

IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

Root cause Capacity exhaustion

IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020

Root cause Software defect

IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the nodeMindTickle · 2020

Root cause DNS or network fault

IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

Root cause Software defect

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

Root cause Faulty deployment

IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019

Root cause Software defect

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

Root cause Configuration change

IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its podsPusher · 2019

Root cause Configuration change

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019

Root cause Operator error

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolverToyota Connected · 2018

Root cause Software defect

IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open filesthredUP · 2018

Root cause Software defect

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

Root cause Capacity exhaustion