Cases2 cases

1 to 2 of 2

1. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran

   Render2023

   Symptoms

   After Point-in-time Recovery (PITR) was released to eligible new Render Postgres databases on 21 April 2023, 50% of new connections to PITR-enabled databases failed while a PITR backup job was running. A backup takes 15 to 45 minutes depending on database size, and existing connections, such as those held through connection pooling, kept working.

   Intermittent connection failuresMisrouted requests

   Trigger

   Point-in-time Recovery was released to eligible new databases on 21 April 2023, and its backup job carried an incorrect networking configuration.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshotsService discovery

   Contributing factor

   –

   Remediation

   • Disabled backups for affected databases until they could be scheduled in low-traffic hours
   • Changed the backup job's network labels so the routing layer no longer picks it up as a database

   Action items

   • Tighten the process for sharing incident updates so problems are not declared resolved too early
   • Use more specific network labels for the database routing layer so ineligible IPs are never added
   • Run a skipped asynchronous job in the next scheduled window instead of immediately
   • Use open source benchmarking software to reproduce issues customers notice

   View case
2. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node

   Render2022

   Symptoms

   At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

   Service outageStorage disconnected or stale mounts

   Trigger

   A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

   Root cause

   Hardware failure

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failure

   Detection

   An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

   Remediation

   • Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes

   Action items

   • Create an alert that fires sooner than the current 15-minute startup alert
   • Make it easier to identify the correct corrective action, and eventually automate it
   • Review incident procedures so the status page reflects reliability accurately
   • Build high availability for Postgres so a primary's node hardware failure causes minimal downtime

   Lessons

   Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

   View case

By root cause2