IL-0289 updown.io · 2019-04-03database

A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down

updown.io runs MongoDB with a primary server, whose data sits on 3 SSDs in RAID5, and a less powerful secondary. When one SSD started to fail at 2:30 am CEST on 3 April 2019, the primary slowed to about 200 times its normal query time but kept answering the heartbeat, so MongoDB never failed over. The founder was asleep and noticed the problem at 8:45 am, checked that the secondary was ready and forced the primary to step down; the website and daemons recovered at 8:56 am, more slowly than usual on the weaker secondary. The founder then found the failing SSD, took it out of the array, which restored the old primary's performance, and asked the hosting provider to replace it. On the following two nights the weaker secondary, now primary, failed again because a daily clamscan antivirus run took memory from MongoDB: on 4 April the kernel's OOM killer killed MongoDB, and on 5 April the server slowed and the DNS automatic failover switched to the disabled primary. On 6 April at 12:48 pm the secondary was replaced with a more powerful server and response times returned to normal. When the provider replaced the SSD, a second SSD had died in the same way, apparently both far past their rated endurance; it was replaced on 16 April and the original server became primary again on 22 April. The script written afterwards to force a step-down when MongoDB is up but unresponsive cut the downtime to under 10 minutes when the third SSD died on 6 November 2019, and in 2020 MongoDB's Storage Node Watchdog, then opened to the community edition, replaced the script.

Severity: –
Scope: The updown.io website and monitoring daemons were almost fully down from 2:30 am to 8:56 am CEST, then ran with lower performance on the less powerful secondary
Users affected: –
Region: –
Time to detect: 6h 15m
Time to resolve: 6h 26m

Symptoms

At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

MongoDB's automatic failover did not react, because the replica set heartbeat check still answered quickly; a secondary server was ready to take over but was never elected primary (MongoDB ticket SERVER-29980). On the primary, atop showed the IO usage ("busy") and average latency ("avgio") of its three disks, and the founder found an IO performance problem.

Symptom High latency Service outage Failover failure

Root cause

Root cause Hardware failure

Component Database Database failover manager

Contributing factor –

Trigger: One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

Detection

The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

Remediation

Forced the primary to step down after making sure the secondary was ready to take over
Took the failing SSD out of the RAID array, which restored the old primary's performance
Asked the hosting provider to replace the SSD
Replaced the less powerful secondary with a more powerful server

Action items

Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
Shut down and delete the old secondary once nothing is left to investigate on it
Make sure the new secondary can run as primary for several days
Test the automatic step-down script once the primary server is healthy again
Provide a proper status page for updown.io
Replace the custom script with MongoDB's Storage Node Watchdog

Lessons

–

Sources

https://updown.io/blog/mongodb-ssd-incident-postmortem · blog · archived copy

References

https://jira.mongodb.org/browse/SERVER-29980

Related cases

IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalledNeon · 2024
same root cause · shared Database, Database failover manager
IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017
same root cause · shared Database, Database failover manager
IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachableSourceHut · 2023
same root cause · shared Database
IL-0370 Render Postgres databases in Frankfurt were down up to 65 minutes as their disks stayed attached to a failed nodeRender · 2022
same root cause · shared Database
IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019
same root cause · shared Database

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0289
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0289/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.