Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Automation amplification

자동화 증폭 · automation-amplified

29 cases · 2017 to 2026

Root cause Configuration change8 Capacity exhaustion7 Software defect6 Dependency failure3 Breaking change in an upgrade3 Faulty deployment1 DNS or network fault1

Component Application database6 Node operating system image6 Node pool5 CoreDNS3 Kubernetes API server

Symptom Service outage14 High error rate11 High latency7 Node NotReady7

Organisation Datadog2 DFDS2 Google Cloud2 Microsoft Azure2 OpenAI

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion

View on the graph

3

Pod scheduling and placement3

Admission webhook2

In-memory cache2

Cloud load balancer2

Configuration templating2

Container runtime2

Horizontal pod autoscaler2

Jobs and CronJobs2

Node launch template2

Container registry and its credentials2

Telemetry agent2

Automated capacity manager1

conntrack table1

Credential broker for pods1

Client-side DNS resolver and cache1

Authoritative DNS servers and records1

GKE control plane1

Liveness and readiness probes1

Ingress controller1

Container memory limit1

Model request routing1

Persistent volumes and stateful sets1

Pod priority and preemption1

Service discovery1

systemd-networkd1

Intermittent connection failures5

Resources deleted5

DNS resolution failures4

Intermittent timeouts3

Node recreation loop3

Autoscaler misbehaviour2

Scale-up failure2

Control plane unresponsive2

Container crash loop2

Database overload2

GPU unavailable2

Image pull failures2

Monitoring outage2

Probe failures2

Control plane lag1

Credentials unavailable1

Sustained high node CPU1

Outbound connection exhaustion1

Stale endpoint routing1

Throughput degradation1

Load imbalance1

2

Amazon Web Services1

IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates

OpenAI · 2026

Root cause Software defect

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

Root cause Software defect

IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loopYonghyun Kim · 2025

Root cause Dependency failure

IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025

Root cause Software defect

IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025

Root cause Software defect

IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

Root cause Breaking change in an upgrade

IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024

Root cause Capacity exhaustion

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

Root cause Capacity exhaustion

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

Root cause Capacity exhaustion

IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

Root cause Breaking change in an upgrade

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022

Root cause Software defect

IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

Root cause Configuration change

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

Root cause Dependency failure

IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018

Root cause Configuration change

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

Root cause Capacity exhaustion