Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

17 to 17 of 17

  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
    Symptoms

    While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

    Service outageFile descriptor exhaustion
    Trigger

    Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

    Root cause
    Software defect
    Component
    Telemetry agent
    Contributing factor
    Unverified assumption
    Detection

    Datadog showed the number of open files on the nodes rising.

    Remediation
    • killed the affected node, brought up a new one and redeployed
    Action items
    • Move away from the log aggregation vendor
    Lessons

    Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

    View case
Check your cluster for this

These 17 cases involve Telemetry agent and Container runtime. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change3 cases

Component
Telemetry agent3Container runtime2Application database1Client-side DNS resolver and cache1Kubernetes API server1Node launch template1Application logging library1Node pool1Terraform1
Contributing factor
Unrepresentative staging2Unverified assumption1Automation amplification1Excessive blast radius1Unreviewed defaults1Cache-masked failure1Limited debug access1Harmful manual action1Monitoring blind spot1No configuration tests1No runbook1No staged rollout1
Organisation
DFDS1MUSINSA1OpenAI1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  3. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023

Faulty deployment3 cases

Component
Telemetry agent3GPU driver1Service discovery1
Contributing factor
Unverified assumption2Excessive blast radius1Missed in review1
Organisation
Airbnb1Google Cloud1Linode1
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  2. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023
  3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019

Capacity exhaustion6 cases

Component
Telemetry agent6etcd2Kubernetes API server2Container memory limit2Application database1Elasticsearch or OpenSearch1
Contributing factor
Monitoring blind spot3Unknown limit2No alert on limit2No resource requests or limits2Unverified assumption1Excessive blast radius1Unreviewed defaults1Ignored early signal1Harmful manual action1No capacity planning1
Organisation
UK Ministry of Justice2Blue Matador1Civis Analytics1DFDS1Palark1
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  2. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  3. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  4. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  5. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  6. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019

DNS or network fault1 case

Component
CoreDNSClient-side DNS resolver and cacheTelemetry agent
Contributing factor
Unverified assumptionUnreviewed defaultsIgnored early signalUnknown limit
Organisation
Adevinta
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023

Software defect3 cases

Component
Telemetry agent3Elasticsearch or OpenSearch1
Contributing factor
Unverified assumption1Unpatched known bug1Limited debug access1Monitoring blind spot1
Organisation
Hyperconnect1PrometheusKube1thredUP1
  1. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
  2. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch
    PrometheusKube2020
  3. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019