Incident Library
Applied filters ComponentTelemetry agentSymptomFile descriptor exhaustionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
    Symptoms

    While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

    Service outageFile descriptor exhaustion
    Trigger

    Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

    Root cause
    Software defect
    Component
    Telemetry agent
    Contributing factor
    Unverified assumption
    Detection

    Datadog showed the number of open files on the nodes rising.

    Remediation
    • killed the affected node, brought up a new one and redeployed
    Action items
    • Move away from the log aggregation vendor
    Lessons

    Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

    View case
Check your cluster for this

This case involves Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Telemetry agent
Contributing factor
Unverified assumption
Organisation
thredUP
  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018