Cases2 cases

1 to 2 of 2

1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case
2. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node

   Render2022

   Symptoms

   At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

   Service outageStorage disconnected or stale mounts

   Trigger

   A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

   Root cause

   Hardware failure

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failure

   Detection

   An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

   Remediation

   • Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes

   Action items

   • Create an alert that fires sooner than the current 15-minute startup alert
   • Make it easier to identify the correct corrective action, and eventually automate it
   • Review incident procedures so the status page reflects reliability accurately
   • Build high availability for Postgres so a primary's node hardware failure causes minimal downtime

   Lessons

   Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

   View case

By root cause1