Cases9 cases

4 to 8 of 9

1. IL-0479 Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signal

   Amazon Web Services2025

   Symptoms

   Machine learning accelerator nodes that a cloud computing platform's automated fleet management had removed from production produced wrong computation results without any error signal. When the same LLM training ran on one of them and on a healthy node, the training loss was identical, but submodule outputs, gradients and model weights differed.

   Training divergence or loss spike

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Monitoring blind spot

   Detection

   The fleet management tests found the nodes: collective communication and compute unit stress tests and a small LLM training run compared with golden values, run after installation and when a node returns from customer use. Training loss showed nothing on most nodes.

   Remediation

   • Kept the nodes that failed the fleet's SDC tests out of production availability
   • Sent the nodes held for the study for repair before returning them to the production pool

   Lessons

   SDC can change training for an unknown period before any loss spike appears, so watching the training loss is not enough. ABFT did not detect it reliably; the authors' possible explanations are that mismatches were too rare and small to exceed floating-point error bounds, that ABFT lowered accelerator utilization and with it the SDC rate, and that SDC may arise outside matrix multiplication; the authors suggest recomputation, for example a shadow data-parallel replica whose gradients are compared with a target rank's during the gradient all-reduce, and point at reducing the sharpness of the loss surface to limit the damage.

   View case
2. IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks

   Meta2024

   Symptoms

   In Meta's RSC-1 and RSC-2 research GPU clusters, certain nodes had above-average rates of job failures. After each failure the same nodes kept attracting new jobs and failing them too, while health checks and repair workflows did not flag them. Meta calls these servers lemon nodes.

   Training job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Monitoring blind spot

   Detection

   Meta's lemon detection flags nodes from their history in per-node signals such as excl_jobid_count, xid_cnt, tickets, out_count and node failure counts, because the periodic health checks and repair workflows did not catch them.

   Remediation

   • Built lemon detection to proactively identify, isolate and replace lemon nodes in the Slurm scheduler
   • Identified 40 faulty nodes, 24 on RSC-1 and 16 on RSC-2

   Lessons

   Finding defective nodes needs historic data, and lemon node detection can improve the completion rate of large jobs by over 30%. A defective node that cannot be removed from capacity leads to restart loops that make large gang-scheduled jobs impossible to run, so recovery from random failures only works once defective nodes are reliably rotated out.

   View case
3. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets

   Korea Ministry of the Interior and Safety2023

   Symptoms

   On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

   Service outagePacket lossHigh latency

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   Monitoring blind spotGray failure

   Remediation

   • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
   • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
   • Swapped out the module of the router that had the fault

   Action items

   • Inspect all old equipment that may have similar port defects
   • Revise the incident handling manual so the public is told about an outage quickly
   • Announce the status of service recovery quickly through various means
   • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
   • Form a standing incident response team to send into major outages
   • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
   • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
   • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
   • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
   • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
   • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
   • Work with manufacturers on a way to find and monitor failures of internal parts earlier

   Lessons

   The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

   View case
4. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case
5. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case

By root cause1