Incident Library
Applied filters Root causeFaulty deploymentComponentGPU driverClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
    Symptoms

    From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

    GPU unavailableControl plane unresponsive
    Trigger

    Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

    Root cause
    Faulty deployment
    Component
    Telemetry agentGPU driver
    Contributing factor
    Missed in reviewUnverified assumption
    Detection

    Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

    Remediation
    • Disabled the monitoring script across the GPU fleet
    • Rebooted the affected nodes
    Action items
    • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them
    Lessons

    A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

    View case
Check your cluster for this

This case involves GPU driver and Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
GPU driverTelemetry agent
Contributing factor
Unverified assumptionMissed in review
Organisation
Linode
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026