Cases9 cases

6 to 9 of 9

1. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets

   Korea Ministry of the Interior and Safety2023

   Symptoms

   On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

   Service outagePacket lossHigh latency

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   Monitoring blind spotGray failure

   Remediation

   • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
   • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
   • Swapped out the module of the router that had the fault

   Action items

   • Inspect all old equipment that may have similar port defects
   • Revise the incident handling manual so the public is told about an outage quickly
   • Announce the status of service recovery quickly through various means
   • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
   • Form a standing incident response team to send into major outages
   • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
   • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
   • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
   • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
   • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
   • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
   • Work with manufacturers on a way to find and monitor failures of internal parts earlier

   Lessons

   The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

   View case
2. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case
3. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case
4. IL-0427 A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed too

   BigScience2021

   Symptoms

   After many hours of training, one job of the tr1-13B Slurm job array crashed at iteration 43680 of 311541 during the backward pass. The next job in the array started by itself and crashed right away, before training began.

   Training job failureGPU unavailable

   Trigger

   One GPU on one node of the allocation developed a hardware problem that produced uncorrectable ECC errors.

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Automation amplificationMonitoring blind spot

   Detection

   Nobody was around to notice the first crash, and the Slurm scheduler started the next job in the array on its own, which crashed too.

   Remediation

   • Contacted the JZ administrators, who confirmed one of the nodes was faulty
   • Continued training with the next job, which was not placed on the faulty node

   Action items

   • Add %N to the Slurm output setting (#SBATCH --output=%x-%j-%N.out) so each node writes its own log file and the node with a corrupt GPU can be identified
   • Exclude a node known to be faulty from the allocation with sbatch --exclude

   Lessons

   Without a way to link an exception to the hostname of the node that raised it, the scheduler can keep placing the training on the faulty node and the training keeps crashing; per-node log files make the faulty node visible so it can be excluded.

   View case

By root cause1