Cases3 cases

1 to 3 of 3

1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case
2. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down

   updown.io2019

   Symptoms

   At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

   High latencyService outageFailover failure

   Trigger

   One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

   Remediation

   • Forced the primary to step down after making sure the secondary was ready to take over
   • Took the failing SSD out of the RAID array, which restored the old primary's performance
   • Asked the hosting provider to replace the SSD
   • Replaced the less powerful secondary with a more powerful server

   Action items

   • Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
   • Shut down and delete the old secondary once nothing is left to investigate on it
   • Make sure the new secondary can run as primary for several days
   • Test the automatic step-down script once the primary server is healthy again
   • Provide a proper status page for updown.io
   • Replace the custom script with MongoDB's Storage Node Watchdog

   View case
3. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

   GoCardless2017

   Symptoms

   At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

   Service outageFailover failure

   Trigger

   The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Overlapping failuresUnverified assumption

   Detection

   GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

   Remediation

   • Powered off the broken primary Postgres node
   • Cleared the Pacemaker error counts with crm resource cleanup, which had no effect
   • Edited recovery.conf on the synchronous replica under maintenance mode several times, which Pacemaker reverted
   • Put the cluster into maintenance mode, configured the synchronous replica as primary and started Postgres by hand
   • Reconfigured the backend applications to connect to the new primary's actual IP address
   • Bootstrapped a third node as an asynchronous replica
   • Added a VIP managed by hand by the infrastructure team for manual failover
   • Verified data integrity and found no corruption
   • Migrated to a new Pacemaker-managed database cluster in a maintenance window and decommissioned the old one

   Action items

   • Express the Backup VIP rule as an INF preference for a replica instead of a -INF preference against the primary
   • Replace the VIPs with proxies on the application servers that route traffic by the state of the cluster
   • Run harsher fault injection tests of the Postgres cluster and consider automation that injects failures continually
   • Consider adding arbitrary restrictions to game day exercises, such as automation that has failed and a problem that cannot be diagnosed, so engineers practise bringing the service back another way

   Lessons

   A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

   View case

By root cause1