Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Monitoring outage

모니터링 중단 · monitoring-down

19 cases · 2019 to 2026

Root cause Capacity exhaustion10 Configuration change2 Software defect2 Breaking change in an upgrade2 Dependency failure1 Operator error1 unknown1

Component Prometheus7 Container memory limit6 Telemetry agent5 kubelet4

Contributing factor Monitoring blind spot8 Unreviewed defaults5 No alert on limit5 Excessive blast radius4

Organisation UK Ministry of Justice6 DFDS2 Barush Mendez1 Fullstaq1 LINE+

Cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025
Root cause Software defect
Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space

View on the graph

Kubernetes API server3

Client-side DNS resolver and cache2

Elasticsearch or OpenSearch2

Admission webhook1

Cloud block storage1

Cloud load balancer1

Container runtime1

Credential broker for pods1

Application database1

Data center power and cooling1

InfiniBand fabric1

Ingress controller1

Node launch template1

Node operating system image1

Cluster authentication provider1

Container registry and its credentials1

Single point of failure4

Harmful manual action3

No resource requests or limits3

Automation amplification2

Ignored early signal2

Unrepresentative staging2

Untested backup1

Configuration not in code1

Change with a delayed effect1

Unpatched known bug1

Limited debug access1

No capacity planning1

No configuration tests1

No safeguard on destructive change1

No staged rollout1

Overlapping failures1

1

Microsoft Azure1

PrometheusKube1

Saifeddine Rajhi1

IL-0042

UK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

Root cause Capacity exhaustion

IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metricsUK Ministry of Justice · 2023

Root cause unknown

IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0041 fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020

Root cause Software defect

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion