Incident Library
Applied filters Root causeFaulty deploymentComponentTelemetry agentFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
    Symptoms

    From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

    GPU unavailableControl plane unresponsive
    Trigger

    Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

    Root cause
    Faulty deployment
    Component
    Telemetry agentGPU driver
    Contributing factor
    Missed in reviewUnverified assumption
    Detection

    Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

    Remediation
    • Disabled the monitoring script across the GPU fleet
    • Rebooted the affected nodes
    Action items
    • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them
    Lessons

    A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

    View case
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
    Symptoms

    An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

    Probe failuresService outage
    Trigger

    A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

    Root cause
    Faulty deployment
    Component
    Telemetry agentService discovery
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

    Action items
    • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check
    Lessons

    Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

    View case
Check your cluster for this

These 2 cases involve Telemetry agent and GPU driver. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
Telemetry agent2GPU driver1Service discovery1
Contributing factor
Unverified assumption2Excessive blast radius1Missed in review1
Organisation
Airbnb1Linode1
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019