Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Pod scheduling and placement

파드 스케줄링과 배치 · pod-scheduling

5 cases · 2017 to 2025

Root cause Capacity exhaustion4 Software defect1

Contributing factor Automation amplification3 Unverified assumption2 Single point of failure2 Excessive blast radius1 Harmful manual action

Symptom High error rate2 High latency2 Intermittent connection failures2 Sustained high node CPU2

Organisation Microsoft1 Moonlight1 PostHog1 Universe1 이언철1

Cases, by severity

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clustersMicrosoft · 2024
Root cause Software defect

View on the graph

1

No alert on limit1

No staged rollout1

Ineffective rollback1

Unrepresentative staging1

Service outage2

Autoscaler misbehaviour1

Container crash loop1

Database overload1

Node recreation loop1

IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

이언철 · 2024

Root cause Capacity exhaustion

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

Root cause Capacity exhaustion