Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Root cause

Configuration change

설정 변경 · config-change

35 cases · 2018 to 2026

Component Cloud load balancer6 Kubernetes API server6 Node pool6 Configuration templating4 Terraform4 AWS VPC CNI4 kubelet3 Container memory limit3 Telemetry agent3 Admission webhook2 Argo CD2 Container runtime2 etcd2 GKE control plane2 HAProxy2 Ingress controller2 Node launch template2 Persistent volumes and stateful sets2 In-memory cache1 CI system1 Cluster dashboard1 CNI plugin1 ConfigMaps and Secrets1 conntrack table1 CoreDNS1 Credential broker for pods1 Application database1 Client-side DNS resolver and cache1 Helm1 Horizontal pod autoscaler1 Jobs and CronJobs1 Linux kernel1 Application logging library1 Node root disk1 Node operating system image1 Node iptables rules1 Pod priority and preemption1 Prometheus1 RBAC and service accounts1 Container registry and its credentials1 Service discovery1

Contributing factor Excessive blast radius11 Monitoring blind spot10 Unverified assumption9 Automation amplification8

Symptom Service outage17 High error rate11 High latency7 Resources deleted7

Organisation DFDS4 OpenAI4 Grafana Labs3 UK Ministry of Justice3 DigitalOcean

Cases

IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loadsGitHub · 2026
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025
IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error ratesOpenAI · 2025
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

View on the graph

Unreviewed defaults7

No configuration tests6

Missed in review6

No safeguard on destructive change4

No staged rollout4

Unrepresentative staging4

Change with a delayed effect3

Limited debug access3

Ignored early signal2

No alert on limit2

No health probes2

Untested backup1

Cache-masked failure1

Harmful manual action1

No resource requests or limits1

Overlapping failures1

Partial rollout1

Single point of failure1

Unpinned version1

Control plane unresponsive4

Intermittent timeouts3

IP address exhaustion3

Deploys failing2

Monitoring outage2

Pods stuck in ContainerCreating2

Probe failures2

Unauthorised access or a foreign workload2

Autoscaler misbehaviour1

Connection pool exhaustion1

Connection resets1

Control plane lag1

Container crash loop1

Database overload1

DNS resolution failures1

Image pull failures1

Intermittent connection failures1

Node disk full1

Sustained high node CPU1

Node recreation loop1

Misrouted requests1

Throughput degradation1

Load imbalance1

2

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

OpenAI · 2024

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration changeGoogle Cloud · 2024

IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023

IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local portsHans H. Kim · 2023

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutesGitLab · 2021

IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every requestAdevinta · 2019

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its podsPusher · 2019

IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018