Incident Library
Applied filters Root causeHardware failureSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

1 to 5 of 6

  1. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets
    Korea Ministry of the Interior and Safety2023
    Symptoms

    On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

    Service outagePacket lossHigh latency
    Root cause
    Hardware failure
    Component
    NIC, cable and switch port link
    Contributing factor
    Monitoring blind spotGray failure
    Remediation
    • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
    • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
    • Swapped out the module of the router that had the fault
    Action items
    • Inspect all old equipment that may have similar port defects
    • Revise the incident handling manual so the public is told about an outage quickly
    • Announce the status of service recovery quickly through various means
    • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
    • Form a standing incident response team to send into major outages
    • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
    • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
    • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
    • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
    • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
    • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
    • Work with manufacturers on a way to find and monitor failures of internal parts earlier
    Lessons

    The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

    View case
  2. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable
    SourceHut2023
    Symptoms

    At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

    Service outage
    Trigger

    remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

    Root cause
    Hardware failure
    Component
    Database
    Contributing factor
    Single point of failureLimited debug access
    Detection

    Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

    Remediation
    • Sent SourceHut's emergency on-call contact in the datacenter area to go on site
    • Moved remilia's disks into the chassis of build worker cirno1.sr.ht and brought it online as remilia
    Action items
    • Make the off-site database copy able to serve as a failover database host
    • Re-prioritize the migration to the EU
    • Set up new database failover
    • Add a new EU build worker to backfill build capacity
    • Record a better hardware inventory for future reference
    View case
  3. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance
    OpenStreetMap Foundation2022
    Symptoms

    From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

    Network unreachableService outage
    Trigger

    Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

    Root cause
    Hardware failure
    Component
    Network switchData center power and cooling
    Contributing factor
    Single point of failureLimited debug accessMonitoring blind spot
    Remediation
    • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC
    Action items
    • Acknowledge ongoing outages faster
    • Add monitoring and alerting of switch PSU and FAN states
    • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
    • Install out-of-band access equipment with dual power feeds in Amsterdam
    • Install redundant uplinks
    • Make site failover easier, less manual and lower risk
    • Introduce an on-call pager system for the operations SRE
    • Expand the SRE team
    • Continue and formalise fail-over tests for power, network, upstream link and service outages
    View case
  4. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node
    Render2022
    Symptoms

    At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

    Service outageStorage disconnected or stale mounts
    Trigger

    A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

    Root cause
    Hardware failure
    Component
    DatabaseCloud block storage
    Contributing factor
    Single point of failure
    Detection

    An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

    Remediation
    • Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes
    Action items
    • Create an alert that fires sooner than the current 15-minute startup alert
    • Make it easier to identify the correct corrective action, and eventually automate it
    • Review incident procedures so the status page reflects reliability accurately
    • Build high availability for Postgres so a primary's node hardware failure causes minimal downtime
    Lessons

    Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

    View case
  5. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down
    updown.io2019
    Symptoms

    At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

    High latencyService outageFailover failure
    Trigger

    One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

    Root cause
    Hardware failure
    Component
    DatabaseDatabase failover manager
    Contributing factor
    –
    Detection

    The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

    Remediation
    • Forced the primary to step down after making sure the secondary was ready to take over
    • Took the failing SSD out of the RAID array, which restored the old primary's performance
    • Asked the hosting provider to replace the SSD
    • Replaced the less powerful secondary with a more powerful server
    Action items
    • Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
    • Shut down and delete the old secondary once nothing is left to investigate on it
    • Make sure the new secondary can run as primary for several days
    • Test the automatic step-down script once the primary server is healthy again
    • Provide a proper status page for updown.io
    • Replace the custom script with MongoDB's Storage Node Watchdog
    View case
Check your cluster for this

These 6 cases involve Database and Database failover manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure6 cases

Component
Database4Database failover manager2Cloud block storage1Data center power and cooling1NIC, cable and switch port link1Network switch1
Contributing factor
Single point of failure3Limited debug access2Monitoring blind spot2Unverified assumption1Gray failure1Overlapping failures1
Organisation
GoCardless1Korea Ministry of the Interior and Safety1OpenStreetMap Foundation1Render1SourceHut1updown.io1
  1. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets
    Korea Ministry of the Interior and Safety2023
  2. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable
    SourceHut2023
  3. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance
    OpenStreetMap Foundation2022
  4. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node
    Render2022
  5. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down
    updown.io2019
  6. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica
    GoCardless2017