Incident Library
Applied filters Root causeHardware failureComponentCloud block storageFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node
    Render2022
    Symptoms

    At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

    Service outageStorage disconnected or stale mounts
    Trigger

    A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

    Root cause
    Hardware failure
    Component
    DatabaseCloud block storage
    Contributing factor
    Single point of failure
    Detection

    An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

    Remediation
    • Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes
    Action items
    • Create an alert that fires sooner than the current 15-minute startup alert
    • Make it easier to identify the correct corrective action, and eventually automate it
    • Review incident procedures so the status page reflects reliability accurately
    • Build high availability for Postgres so a primary's node hardware failure causes minimal downtime
    Lessons

    Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

    View case
Check your cluster for this

This case involves Cloud block storage and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
Cloud block storageDatabase
Contributing factor
Single point of failure
Organisation
Render
  1. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node
    Render2022