Cases6 cases

1 to 5 of 6

1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case
2. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable

   SourceHut2023

   Symptoms

   At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

   Service outage

   Trigger

   remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

   Root cause

   Hardware failure

   Component

   Database

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

   Remediation

   • Sent SourceHut's emergency on-call contact in the datacenter area to go on site
   • Moved remilia's disks into the chassis of build worker cirno1.sr.ht and brought it online as remilia

   Action items

   • Make the off-site database copy able to serve as a failover database host
   • Re-prioritize the migration to the EU
   • Set up new database failover
   • Add a new EU build worker to backfill build capacity
   • Record a better hardware inventory for future reference

   View case
3. IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed node

   Render2022

   Symptoms

   At 08:30 UTC on 23 May 2022, every user Postgres database on one node in Render's Frankfurt region went completely down after the node, supplied by Render's external cloud provider, had a hardware issue and failed its status check. The database instances could not start on a healthy node because their disks were still attached to the failed node.

   Service outageStorage disconnected or stale mounts

   Trigger

   A node supplied by Render's external cloud provider had a hardware issue and failed its status check at 08:30 UTC on 23 May 2022.

   Root cause

   Hardware failure

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failure

   Detection

   An alert that fires when databases have been stuck in one state while starting up for at least 15 minutes paged the on-call engineer automatically at 08:57 UTC.

   Remediation

   • Manually terminated the failed node at about 09:27 UTC so the disks could attach to healthy nodes

   Action items

   • Create an alert that fires sooner than the current 15-minute startup alert
   • Make it easier to identify the correct corrective action, and eventually automate it
   • Review incident procedures so the status page reflects reliability accurately
   • Build high availability for Postgres so a primary's node hardware failure causes minimal downtime

   Lessons

   Hardware issues are rare but expected, so Render's mitigations aim at detecting them sooner and shortening the time to resolve them.

   View case
4. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only

   Wikimedia Foundation2019

   Symptoms

   At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

   Writes rejectedReplication lagHigh latency

   Trigger

   The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase replication

   Contributing factor

   Pages not deliveredHarmful manual action

   Detection

   Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

   Remediation

   • Power-cycled db1075 from the iDRAC after finding the BBU problem
   • Let MySQL finish InnoDB recovery on db1075
   • Removed read_only=ON from db1075 by hand
   • Ran puppet by hand a second time so pt-heartbeat started

   Action items

   • Implement or refactor a script that moves replicas when the primary is unavailable
   • Fix the MediaWiki heartbeat model so pt-heartbeat does not use a super-user, avoids a single point of failure and follows the real primary without depending on puppet
   • Decide what to do with the batch of hosts that already had BBU issues
   • Buy a new BBU for db1075
   • Remove db1075 from the primary role
   • Address MediaWiki spam while the database is read-only or the primary is unavailable
   • Make primary database hosts page on HOST DOWN
   • Track hardware errors better in Netbox

   Lessons

   A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

   View case
5. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down

   updown.io2019

   Symptoms

   At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

   High latencyService outageFailover failure

   Trigger

   One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

   Remediation

   • Forced the primary to step down after making sure the secondary was ready to take over
   • Took the failing SSD out of the RAID array, which restored the old primary's performance
   • Asked the hosting provider to replace the SSD
   • Replaced the less powerful secondary with a more powerful server

   Action items

   • Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
   • Shut down and delete the old secondary once nothing is left to investigate on it
   • Make sure the new secondary can run as primary for several days
   • Test the automatic step-down script once the primary server is healthy again
   • Provide a proper status page for updown.io
   • Replace the custom script with MongoDB's Storage Node Watchdog

   View case

By root cause1