Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Sustained high node CPU

노드 CPU 고부하 · node-high-cpu

5 cases · 2019 to 2025

Root cause Capacity exhaustion3 Configuration change1 unknown1

Component Application database2 Node pool2 Pod scheduling and placement2 In-memory cache1

Contributing factor Unverified assumption3 Excessive blast radius2 Single point of failure2 Automation amplification1

Organisation 2i2c1 Fyra Labs1 JW Player1 Moonlight1 PostHog1

Cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every nodeFyra Labs · 2023
Root cause unknown
Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

View on the graph

Cloud load balancer1

Cluster dashboard1

Container runtime1

Container memory limit1

Multus meta-CNI1

Unreviewed defaults1

Ignored early signal1

Harmful manual action1

Monitoring blind spot1

No alert on limit1

No resource requests or limits1

No staged rollout1

Ineffective rollback1

Unrepresentative staging1

IL-0094

2i2c · 2020

Root cause Capacity exhaustion

IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

Root cause Configuration change

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion