Incident Library
Applied filters ComponentTelemetry agentFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
    Symptoms

    While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

    Service outageFile descriptor exhaustion
    Trigger

    Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

    Root cause
    Software defect
    Component
    Telemetry agent
    Contributing factor
    Unverified assumption
    Detection

    Datadog showed the number of open files on the nodes rising.

    Remediation
    • killed the affected node, brought up a new one and redeployed
    Action items
    • Move away from the log aggregation vendor
    Lessons

    Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

    View case
Check your cluster for this

These 6 cases involve Telemetry agent and Container runtime. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change1 case

Component
Container runtimeApplication databaseApplication logging libraryTelemetry agent
Contributing factor
Unverified assumptionExcessive blast radius
Organisation
MUSINSA
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026

Faulty deployment2 cases

Component
Telemetry agent2GPU driver1Service discovery1
Contributing factor
Unverified assumption2Excessive blast radius1Missed in review1
Organisation
Airbnb1Linode1
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019

DNS or network fault1 case

Component
CoreDNSClient-side DNS resolver and cacheTelemetry agent
Contributing factor
Unverified assumptionUnreviewed defaultsIgnored early signalUnknown limit
Organisation
Adevinta
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023

Capacity exhaustion1 case

Component
etcdKubernetes API serverTelemetry agent
Contributing factor
Unverified assumptionUnknown limitMonitoring blind spotNo capacity planning
Organisation
Civis Analytics
  1. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019

Software defect1 case

Component
Telemetry agent
Contributing factor
Unverified assumption
Organisation
thredUP
  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018