Cases2 cases
1 to 2 of 2
- Symptoms
On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.
- Root cause
- Component
- Contributing factor
- Remediation
- Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
- Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
- Swapped out the module of the router that had the fault
- Action items
- Inspect all old equipment that may have similar port defects
- Revise the incident handling manual so the public is told about an outage quickly
- Announce the status of service recovery quickly through various means
- Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
- Form a standing incident response team to send into major outages
- Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
- Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
- Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
- Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
- Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
- Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
- Work with manufacturers on a way to find and monitor failures of internal parts earlier
- Lessons
The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.
- Symptoms
From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.
- Trigger
Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.
- Root cause
- Component
- Contributing factor
- Remediation
- Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC
- Action items
- Acknowledge ongoing outages faster
- Add monitoring and alerting of switch PSU and FAN states
- Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
- Install out-of-band access equipment with dual power feeds in Amsterdam
- Install redundant uplinks
- Make site failover easier, less manual and lower risk
- Introduce an on-call pager system for the operations SRE
- Expand the SRE team
- Continue and formalise fail-over tests for power, network, upstream link and service outages
These 2 cases involve Data center power and cooling and NIC, cable and switch port link. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.