Cases2 cases

1 to 2 of 2

1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions

   Linode2026

   Symptoms

   From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

   GPU unavailableControl plane unresponsive

   Trigger

   Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

   Root cause

   Faulty deployment

   Component

   Telemetry agentGPU driver

   Contributing factor

   Missed in reviewUnverified assumption

   Detection

   Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

   Remediation

   • Disabled the monitoring script across the GPU fleet
   • Rebooted the affected nodes

   Action items

   • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them

   Lessons

   A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

   View case
2. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case

By root cause2