Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Root cause

Capacity exhaustion

용량 한계 · capacity

51 cases · 2017 to 2026

Component Container memory limit12 Node pool11 Kubernetes API server9 CoreDNS8 Application database6 kubelet6 Prometheus6 Telemetry agent6 Client-side DNS resolver and cache4 etcd4 Pod scheduling and placement4 In-memory cache3 Horizontal pod autoscaler3 Ingress controller3 VPC subnet or pod address range3 Cluster autoscaler2 conntrack table2 Container runtime2 Container CPU limit2 Karpenter2 kube-dns2 kube-proxy2 Linux kernel2 Node root disk2 Persistent volumes and stateful sets2 Shared network filesystem2 AWS VPC CNI2 Admission webhook1 CI system1 Cilium1 Cloud controller manager1 Credential broker for pods1 Elasticsearch or OpenSearch1 GKE control plane1 HAProxy1 Helm1 Istio1 Jobs and CronJobs1 Language runtime settings1 NAT gateway1 Cluster authentication provider1 Container registry and its credentials1 Service discovery1 Training data loader1 VPC DNS resolver1

Contributing factor Unreviewed defaults17 Monitoring blind spot13 Unknown limit12 Single point of failure12

Symptom OOMKilled13 Service outage13 High error rate12 High latency11

Organisation UK Ministry of Justice5 2i2c3 Zalando3 Backpackr2 loveholidays

Cases

IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026
IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storageLablup Inc · 2026
IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026

View on the graph

Automation amplification7

Excessive blast radius7

Harmful manual action7

Unverified assumption6

No capacity planning6

No alert on limit5

No resource requests or limits5

Ignored early signal3

Unrepresentative staging3

Limited debug access2

No staged rollout2

Ineffective rollback2

Change during peak hours1

Overlapping failures1

DNS resolution failures10

Monitoring outage10

Control plane unresponsive7

Container crash loop6

Intermittent timeouts5

Control plane lag4

Database overload4

Intermittent connection failures4

IP address exhaustion3

Sustained high node CPU3

Probe failures3

Autoscaler misbehaviour2

CPU throttling2

Image pull failures2

Throughput degradation2

Scale-up failure1

Cloud API rate limit exhausted1

Connection resets1

Credentials unavailable1

Deploys failing1

Node disk full1

Node recreation loop1

Load imbalance1

Persistent volume full1

2

Civis Analytics1

Onkar Raghunath Shelke1

Saifeddine Rajhi1

IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025

IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1

IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolutionMercari · 2025 · sev1

IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025

IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025

IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025

IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024

IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024

IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024

IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023

IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023

IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worseGoogle Cloud · 2022

IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019

IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017