Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Prometheus

Prometheus · prometheus

9 cases · 2019 to 2024

Root cause Capacity exhaustion6 Configuration change1 Operator error1 unknown1

Contributing factor Unreviewed defaults3 Monitoring blind spot3 Excessive blast radius2 Unknown limit2

Symptom Monitoring outage7 OOMKilled4 Probe failures4 High latency2

Organisation UK Ministry of Justice4 2i2c1 LINE+1 NU.nl1 Saifeddine Rajhi

Cases, by severity

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024
Root cause Capacity exhaustion
IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024
Root cause Capacity exhaustion
IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics

View on the graph

No alert on limit2

No resource requests or limits2

No capacity planning1

No safeguard on destructive change1

CPU throttling1

Container crash loop1

High error rate1

Intermittent timeouts1

Resources deleted1

Service outage1

Stale endpoint routing1

1

UK Ministry of Justice · 2023

Root cause unknown

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023

Root cause Capacity exhaustion

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change