Cases1 case

1 to 1 of 1

1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case

By root cause1