Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Database overload

데이터베이스 과부하 · database-overloaded

7 cases · 2022 to 2025

Root cause Capacity exhaustion4 Configuration change1 Software defect1 Breaking change in an upgrade1

Component Application database5 In-memory cache2 Ingress controller2 Cloud load balancer1

Contributing factor Retry storm4 Automation amplification2 No staged rollout2 Unverified assumption1

Organisation GitLab1 Honeycomb1 Hugging Face1 LINE+1 OpenAI1

Cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024
Root cause Breaking change in an upgrade

View on the graph

Cluster autoscaler1

conntrack table1

Kubernetes API server1

Container memory limit1

Pod scheduling and placement1

Service discovery1

Telemetry agent1

Excessive blast radius1

Unreviewed defaults1

Change with a delayed effect1

No alert on limit1

Overlapping failures1

Ineffective rollback1

Single point of failure1

Unrepresentative staging1

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

Root cause Capacity exhaustion

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

Root cause Capacity exhaustion