Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Telemetry agent

텔레메트리 에이전트 · telemetry-agent

17 cases · 2018 to 2026

Root cause Capacity exhaustion6 Configuration change3 Faulty deployment3 Software defect3 Dependency failure1 DNS or network fault1

Contributing factor Unverified assumption6 Monitoring blind spot5 Excessive blast radius4 Unreviewed defaults3

Symptom Monitoring outage5 Control plane unresponsive4 High error rate4 Service outage4

Organisation DFDS2 UK Ministry of Justice2 Adevinta1 Airbnb1 Blue Matador

Cases, by severity

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
Root cause Configuration change
IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1
Root cause Dependency failure
With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts

View on the graph

Automation amplification2

Ignored early signal2

Unpatched known bug2

Limited debug access2

Harmful manual action2

No alert on limit2

No resource requests or limits2

Unrepresentative staging2

Cache-masked failure1

No capacity planning1

No configuration tests1

No staged rollout1

Missed in review1

Single point of failure1

DNS resolution failures3

Container crash loop2

Connection pool exhaustion1

Control plane lag1

Credentials unavailable1

Database overload1

File descriptor exhaustion1

GPU unavailable1

Intermittent timeouts1

Probe failures1

Resources deleted1

Load imbalance1

1

Civis Analytics1

PrometheusKube1

IL-0179

MUSINSA · 2026

Root cause Configuration change

IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026

Root cause Faulty deployment

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customersGoogle Cloud · 2023

Root cause Faulty deployment

IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

Root cause DNS or network fault

IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its typeHyperconnect · 2023

Root cause Software defect

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0041 fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020

Root cause Software defect

IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

Root cause Faulty deployment

IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open filesthredUP · 2018

Root cause Software defect