Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Probe failures

프로브 실패 · readiness-probe-failures

16 cases · 2018 to 2026

Root cause Capacity exhaustion3 Software defect3 Configuration change2 Dependency failure2 Faulty deployment2 unknown2 Cloud provider outage1 Operator error1

Component Container memory limit4 Prometheus4 kubelet3 In-memory cache2 Application database

Contributing factor Unreviewed defaults6 Unknown limit4 Unverified assumption3 Excessive blast radius3

Organisation OpenAI2 UK Ministry of Justice2 AdGuard VPN1 Airbnb1 DevOps Hof

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025
Root cause Configuration change
A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

View on the graph

2

Ingress controller2

Service discovery2

Admission webhook1

Cloud load balancer1

Configuration templating1

Container CPU limit1

Liveness and readiness probes1

Kubernetes API server1

Model request routing1

Node operating system image1

Persistent volumes and stateful sets1

Telemetry agent1

Monitoring blind spot3

Automation amplification2

Ignored early signal2

Health check tied to a dependency2

Single point of failure2

Change during peak hours1

Change with a delayed effect1

Unpatched known bug1

Harmful manual action1

No alert on limit1

No capacity planning1

No configuration tests1

No health probes1

No resource requests or limits1

Unrepresentative staging1

1

Microsoft Azure1

Saifeddine Rajhi1

Tetrate (unnamed U.S. federal customer)1

IL-0187

Yonghyun Kim · 2025

Root cause Dependency failure

IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiodTetrate (unnamed U.S. federal customer) · 2025 · sev1

Root cause Dependency failure

IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layerOpenAI · 2025

Root cause Cloud provider outage

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

Root cause Capacity exhaustion

IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

Root cause Software defect

IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metricsUK Ministry of Justice · 2023

Root cause unknown

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

Root cause Configuration change

IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

Root cause Software defect

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

Root cause Faulty deployment

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019

Root cause Operator error

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown