Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

High error rate

오류율 상승 · high-error-rate

52 cases · 2017 to 2026

Root cause Capacity exhaustion12 Software defect12 Configuration change11 Breaking change in an upgrade5 Cloud provider outage3 Faulty deployment3 DNS or network fault3 Operator error2 Certificate or credential expiry1

Component Application database12 Kubernetes API server8 Cloud load balancer7 Ingress controller7

Contributing factor Monitoring blind spot13 Automation amplification11 Excessive blast radius10 Unverified assumption8

Organisation OpenAI10 GitLab3 Google Cloud3 Grafana Labs3 Zalando

Cases

IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
Root cause Configuration change
IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment

View on the graph

Container memory limit4

Telemetry agent4

In-memory cache3

conntrack table3

Client-side DNS resolver and cache3

Admission webhook2

Horizontal pod autoscaler2

Model request routing2

Persistent volumes and stateful sets2

Pod scheduling and placement2

Pod termination2

AKS control plane1

Automated capacity manager1

Cluster autoscaler1

Configuration templating1

ConfigMaps and Secrets1

Container runtime1

Container CPU limit1

Credential broker for pods1

Authoritative DNS servers and records1

GKE control plane1

Model inference server1

Application logging library1

Managed Kubernetes control plane and provider API1

Node root disk1

Node operating system image1

Pod priority and preemption1

Serverless function platform1

Service discovery1

systemd-networkd1

Web application firewall rules1

Unreviewed defaults8

Single point of failure7

No safeguard on destructive change6

Change with a delayed effect5

No staged rollout5

No alert on limit4

Unrepresentative staging4

Ignored early signal3

Unpatched known bug3

Limited debug access3

Overlapping failures3

Change during peak hours2

Harmful manual action2

No capacity planning2

No configuration tests2

Partial rollout2

Health check tied to a dependency2

Ineffective rollback2

Missed in review2

Untested backup1

Configuration not in code1

Cache-masked failure1

No resource requests or limits1

Deferred upgrades1

3

Amazon Web Services2

Microsoft Azure2

Civis Analytics1

Lotte E-commerce1

UK Ministry of Justice1

IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026

Root cause Breaking change in an upgrade

IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loadsGitHub · 2026

Root cause Configuration change

IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error ratesOpenAI · 2026

Root cause Software defect

IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthyYonghyun Kim · 2025

Root cause Faulty deployment

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

Root cause Capacity exhaustion

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

Root cause Software defect

IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errorsLinode · 2025

Root cause Certificate or credential expiry

IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error ratesOpenAI · 2025

Root cause Configuration change

IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managedOpenAI · 2025

Root cause Faulty deployment

IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025

Root cause Breaking change in an upgrade

IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025

Root cause Capacity exhaustion

IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hoursGitHub · 2025

Root cause Breaking change in an upgrade

IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layerOpenAI · 2025

Root cause Cloud provider outage

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024

Root cause Capacity exhaustion

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

Root cause Configuration change

IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

Root cause Breaking change in an upgrade

IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024

Root cause Software defect

IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

Root cause Software defect

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

Root cause Capacity exhaustion

IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023

Root cause Cloud provider outage

IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

Root cause DNS or network fault

IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioningAmazon Web Services · 2023

Root cause Software defect

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

Root cause Operator error

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worseGoogle Cloud · 2022

Root cause Capacity exhaustion

IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022

Root cause Software defect

IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

Root cause Capacity exhaustion

IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

Root cause Configuration change

IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

Root cause Software defect

IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

Root cause Configuration change

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect

IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020

Root cause Software defect

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

Root cause Capacity exhaustion

IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its podsPusher · 2019

Root cause Configuration change

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

Root cause Capacity exhaustion

IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018

Root cause Software defect

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect