Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Application database

애플리케이션 데이터베이스 · database

18 cases · 2019 to 2026

Root cause Capacity exhaustion6 Cloud provider outage4 Software defect3 Configuration change1 Dependency failure1 Faulty deployment1 Operator error1 Breaking change in an upgrade1

Contributing factor Automation amplification6 Unverified assumption5 Excessive blast radius5 Retry storm5

Symptom High error rate12 Service outage9 High latency6 Database overload5

Organisation GitLab2 Microsoft Azure2 OpenAI2 Amazon Web Services1 Datadog

Cases, by severity

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1
Root cause Capacity exhaustion
IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
Root cause Configuration change

View on the graph

Single point of failure5

No staged rollout3

Harmful manual action2

Monitoring blind spot2

Overlapping failures2

Health check tied to a dependency2

Change during peak hours1

Unpatched known bug1

No alert on limit1

No resource requests or limits1

No safeguard on destructive change1

Ineffective rollback1

Unrepresentative staging1

Container crash loop4

Intermittent connection failures3

Control plane unresponsive2

DNS resolution failures2

Intermittent timeouts2

Sustained high node CPU2

Probe failures2

Scale-up failure1

Connection pool exhaustion1

Image pull failures1

Monitoring outage1

Outbound connection exhaustion1

Resources deleted1

1

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours

Microsoft Azure · 2026

Root cause Faulty deployment

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

Root cause Capacity exhaustion

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

Root cause Software defect

IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hangLinode · 2025

Root cause Dependency failure

IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layerOpenAI · 2025

Root cause Cloud provider outage

IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

Root cause Breaking change in an upgrade

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023

Root cause Cloud provider outage

IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

Root cause Operator error

IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

Root cause Capacity exhaustion

IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

Root cause Cloud provider outage

IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

Root cause Software defect

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion