Cases2 cases

1 to 2 of 2

1. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case
2. IL-0427 A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed too

   BigScience2021

   Symptoms

   After many hours of training, one job of the tr1-13B Slurm job array crashed at iteration 43680 of 311541 during the backward pass. The next job in the array started by itself and crashed right away, before training began.

   Training job failureGPU unavailable

   Trigger

   One GPU on one node of the allocation developed a hardware problem that produced uncorrectable ECC errors.

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Automation amplificationMonitoring blind spot

   Detection

   Nobody was around to notice the first crash, and the Slurm scheduler started the next job in the array on its own, which crashed too.

   Remediation

   • Contacted the JZ administrators, who confirmed one of the nodes was faulty
   • Continued training with the next job, which was not placed on the faulty node

   Action items

   • Add %N to the Slurm output setting (#SBATCH --output=%x-%j-%N.out) so each node writes its own log file and the node with a corrupt GPU can be identified
   • Exclude a node known to be faulty from the allocation with sbatch --exclude

   Lessons

   Without a way to link an exception to the hostname of the node that raised it, the scheduler can keep placing the training on the faulty node and the training keeps crashing; per-node log files make the faulty node visible so it can be excluded.

   View case

By root cause1