Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Ineffective rollback

롤백 무효 · revert-did-not-restore

4 cases · 2017 to 2025

Root cause Capacity exhaustion2 Software defect1 Breaking change in an upgrade1

Component Node pool2 In-memory cache1 Cluster autoscaler1 ConfigMaps and Secrets1

Symptom Container crash loop2 High error rate2 Service outage2 Scale-up failure1

Organisation Monzo1 Neon1 PostHog1 Wealthy1

Cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1
Root cause Breaking change in an upgrade

View on the graph

Application database1

Client-side DNS resolver and cache1

GKE control plane1

Kubernetes API server1

Persistent volumes and stateful sets1

Pod scheduling and placement1

VPC subnet or pod address range1

Database overload1

DNS resolution failures1

IP address exhaustion1

Sustained high node CPU1

Stale endpoint routing1

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

Neon · 2025

Root cause Capacity exhaustion

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect