Incident Library
Applied filters Root causeSoftware defectFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
    Symptoms

    While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

    Service outageFile descriptor exhaustion
    Trigger

    Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

    Root cause
    Software defect
    Component
    Telemetry agent
    Contributing factor
    Unverified assumption
    Detection

    Datadog showed the number of open files on the nodes rising.

    Remediation
    • killed the affected node, brought up a new one and redeployed
    Action items
    • Move away from the log aggregation vendor
    Lessons

    Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

    View case
Check your cluster for this

These 8 cases involve Ingress controller and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect8 cases

Component
Ingress controller2Node operating system image2Argo CD1Configuration templating1Container CPU limit1Application database1Client-side DNS resolver and cache1GPU driver1HAProxy1Karpenter1Linux kernel1Container memory limit1Persistent volumes and stateful sets1Pod termination1Telemetry agent1
Contributing factor
Unverified assumption8Automation amplification1Unreviewed defaults1Ignored early signal1Unpatched known bug1Harmful manual action1No safeguard on destructive change1No staged rollout1Single point of failure1Unrepresentative staging1
Organisation
Argo CD users1Buffer1Inflab1Palark1Ravelin1Skyscanner1thredUP1Toyota Connected1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  3. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  4. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  5. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
  6. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
  7. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  8. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018