Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Unpatched known bug

미패치 알려진 버그 · known-bug-unpatched

9 cases · 2018 to 2025

Root cause Software defect5 Cloud provider outage1 Dependency failure1 DNS or network fault1 Breaking change in an upgrade1

Component Linux kernel4 Node operating system image3 Cloud load balancer2 Container CPU limit2

Symptom Node NotReady5 High latency4 High error rate3 Service outage3

Organisation AirMap1 Altinity1 Buffer1 Grafana Labs1 Omio1

Cases

IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025
Root cause Breaking change in an upgrade
IL-0041 fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020
Root cause Software defect
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

View on the graph

Telemetry agent2

conntrack table1

Container runtime1

Credential broker for pods1

Application database1

Elasticsearch or OpenSearch1

Kubernetes API server1

Node root disk1

Persistent volumes and stateful sets1

Service discovery1

CPU throttling2

Intermittent timeouts2

Container crash loop1

Credentials unavailable1

DNS resolution failures1

Intermittent connection failures1

Monitoring outage1

Probe failures1

Load imbalance1

PrometheusKube1

Buffer · 2020

Root cause Software defect

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault

IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019

Root cause Software defect

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

Root cause Dependency failure

IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018

Root cause Software defect