IL-0370 Render · 2022-05-23database

Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node

A node from Render's external cloud provider in Frankfurt failed its status check because of a hardware issue. The Postgres database instances on it had to start on a healthy node, but their disks were still attached to the failed node, so the instances could not start and all user databases on that node were fully down. The on-call engineer was paged automatically at 08:57 UTC, found the disk attachment problem and terminated the failed node by hand at about 09:27 UTC, 30 minutes after the page. The disks could then attach to healthy nodes, and all database instances were back online by 09:35 UTC. The status page incident was opened late and recorded the outage as 19 minutes of partial downtime; Render could not edit the incident and corrected the uptime figure by hand to the full 1 hour and 5 minutes. Render was working on high availability for Postgres, expected around Q4 2022, so that databases opting in would see minimal downtime when the primary's node has a hardware issue.

Severity: –
Scope: All user Postgres databases on the failed node had a complete outage
Users affected: –
Region: Frankfurt
Time to detect: 27m
Time to resolve: 1h 5m

Symptoms

At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

The alert that paged the on-call engineer fires only after databases have been stuck in one state while starting up for at least 15 minutes, so the page came at 08:57 UTC, 27 minutes into the outage. The status page incident was created 46 minutes into the outage and showed 19 minutes of partial downtime instead of the 1 hour and 5 minutes of total downtime.

Symptom Service outage Storage disconnected or stale mounts

Root cause

Root cause Hardware failure

Component Database Cloud block storage

Contributing factor Single point of failure

Trigger: A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

Detection

An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

Remediation

Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes

Action items

Create an alert that fires sooner than the current 15-minute startup alert
Make it easier to identify the correct corrective action, and eventually automate it
Review incident procedures so the status page reflects reliability accurately
Build high availability for Postgres so a primary's node hardware failure causes minimal downtime

Lessons

Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

Sources

https://status.render.com/incidents/tk4lms6lbjfh · official-status · archived copy

Related cases

IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalledNeon · 2024
same root cause · shared Database
IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachableSourceHut · 2023
same root cause · shared Database
IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019
same root cause · shared Database
IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went downupdown.io · 2019
same root cause · shared Database
IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017
same root cause · shared Database

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Cloud block storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0370
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0370/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.