Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Missed in review

리뷰 누락 · review-missed

7 cases · 2019 to 2026

Root cause Configuration change6 Faulty deployment1

Component Argo CD2 Configuration templating2 CI system1 Cloud load balancer1 CoreDNS1

Symptom Service outage4 Resources deleted3 High error rate2 Pod eviction2

Organisation DFDS1 Grafana Labs1 InfluxData1 Linode1 Skyscanner1

Cases

IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026
Root cause Faulty deployment
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

View on the graph

GKE control plane1

Ingress controller1

Node launch template1

Node root disk1

Persistent volumes and stateful sets1

Pod priority and preemption1

Telemetry agent1

Control plane unresponsive1

GPU unavailable1

Image pull failures1

IP address exhaustion1

Node disk full1

Pods stuck in ContainerCreating1

UK Ministry of Justice1

IL-0009

InfluxData · 2022

Root cause Configuration change

IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

Root cause Configuration change

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

Root cause Configuration change