IL-0306 Neon · 2024-08-14databaseaws
A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled
Neon stores databases on two services, safekeepers and pageservers. Pageservers ingest the Postgres WAL stored by the safekeepers and cache and index it at 8KiB page granularity for Neon compute instances to read. Durable data lives in S3, so losing a pageserver loses no data, but its projects become unavailable whenever Postgres needs a page, until they are attached to another pageserver. Pageservers run on EC2 instances. AWS usually warns before a hardware failure, but on 2024-08-14 the instance behind pageserver-7 in us-east-1 failed without warning at 17:28.
Alerts reached the team at 17:38, the incident was declared at 17:41, and the node was identified as bad at 17:45. At 17:52 the team decided to migrate the projects on it to other pageservers in the region, and the migration started at 17:58. About 30 minutes passed between the failure and the start of the migration, because a human had to make the decision, prompted by alerts that take several minutes to notice missing telemetry. The semi-manual migration script took about 40 minutes for the 17037 projects, and about 20 projects with operations in flight that blocked migration took about another hour of manual work. All projects were available at 19:41. No data was lost, there was no split brain, and because every project already had a secondary pageserver location prepared, the target nodes had capacity and warm caches.
Neon's control plane runs operations to completion, one at a time per project, which assumes the systems underneath are always available and does not hold for physical nodes that can fail. Neon had built a storage controller for this: it schedules projects onto pageservers with a reconciliation loop, treats every call to a pageserver as fallible, detects failed pageservers with its own heartbeats, and moves their work to other nodes in seconds. It had been in production since May 2024 and managed all projects above 64GiB, but most smaller projects were not yet on it.
- Severity
- –
- Scope
- Postgres databases of affected projects stalled on reads not in their buffer cache; most affected customers lost access for about 30 to 60 minutes, a smaller number for about 2 hours
- Users affected
- about 0.4% of customer projects in us-east-1
- Region
- us-east-1
- Time to detect
- 10m
- Time to resolve
- 2h 13m
Symptoms
At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.
Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the error rate of the control plane API rose, and user operations such as starting endpoints and creating timelines got stuck. The engineers found that the node had no metrics and no logs and could not be reached over SSH.
About 0.4% of customer projects in us-east-1 were affected. The 17037 projects assigned to the failed pageserver were moved to other pageservers from 17:58, and 99.5% were done by 18:40, but about 20 projects in unusual states needed manual work. All projects were available at 19:41. Most affected customers lost access for about 30 to 60 minutes, and a smaller number for about 2 hours.
Root cause
Root cause Hardware failure
- Trigger
- The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.
Detection
Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.
Remediation
- Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
- Brought back by hand about 20 projects whose in-flight operations blocked the migration
Action items
- Move all paying customers onto the storage controller sooner than planned
- Finish moving the existing pageserver infrastructure under the storage controller
Lessons
Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.
Sources
Related cases
IL-0289A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went downupdown.io · 2019same root cause · shared Database, Database failover manager
IL-0219GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017same root cause · shared Database, Database failover manager
IL-0508All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachableSourceHut · 2023same root cause · shared Database
IL-0370Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed nodeRender · 2022same root cause · shared Database
IL-0323A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0306
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0306/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.