Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No runbook

런북 부재 · no-runbook

10 cases · 2021 to 2026

Root cause Configuration change3 Dependency failure2 Faulty deployment2 Capacity exhaustion1 Operator error1 Software defect1

Component Persistent volumes and stateful sets3 Terraform3 Application database2 InfiniBand fabric2

Symptom Service outage7 Resources deleted4 High error rate3 High latency2

Organisation 2i2c1 Amazon Web Services1 CircleCI1 DFDS1 InfluxData

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure

View on the graph

Ingress controller2

Cloud block storage1

In-memory cache1

Cloud controller manager1

Cloud load balancer1

Configuration templating1

Container runtime1

Data center power and cooling1

Authoritative DNS servers and records1

GPU hardware and memory1

Jobs and CronJobs1

Node launch template1

Object storage1

Telemetry agent1

Intermittent connection failures2

Monitoring outage2

Scale-up failure1

Cloud API rate limit exhausted1

Control plane unresponsive1

DNS resolution failures1

GPU unavailable1

Intermittent timeouts1

Outbound connection exhaustion1

Probe failures1

Training job failure1

1

Microsoft Azure1

UK Ministry of Justice1

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

Amazon Web Services · 2025

Root cause Software defect

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

Root cause Operator error

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

Root cause Capacity exhaustion

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

Root cause Configuration change

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure

IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

Root cause Faulty deployment