IL-0489 Magic Pages · 2025-01-15databasekubernetes
Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
Magic Pages runs a three-node high-availability MySQL cluster on Kubernetes, with MySQL Router directing traffic to the database nodes. After one node failed, rejoining it failed with missing-metadata errors, and scaling down to one node to rebuild the metadata spread the errors to the second node. Investigation found that the Kubernetes Persistent Volume Claims appeared to exist but could not provision new volumes for the database nodes that had to be rebuilt.
Recovery took three steps. A restart of the storage server, which took a few seconds, fixed the PVC provisioning. The MySQL nodes then needed about 45 minutes to restore and sync with each other, which the rebuild of the cluster metadata required. Finally the corrupted metadata was cleaned up and MySQL Router was redeployed. No data was lost. The post names the cluster metadata as a single point of failure despite the redundant setup.
- Severity
- –
- Scope
- Magic Pages customer websites, which depend on the MySQL cluster, from about 09:55 to 11:36 CET
- Users affected
- Magic Pages customers
- Region
- –
- Time to detect
- –
- Time to resolve
- 1h 41m
Symptoms
On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.
Earlier that day one of the three MySQL nodes had gone down, and the other two had kept serving without interruption. Because errors about missing metadata usually point to cluster configuration problems, the first attempt was to scale the cluster down to a single node and rebuild the metadata. That produced similar errors on the second node. Magic Pages' status page, which is set up to detect failing customer websites, did not notice the outage straight away.
Symptom Service outage
Root cause
Root cause Dependency failure
- Trigger
- One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.
Detection
–
Remediation
- Restarted the storage server to fix PVC provisioning
- Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
- Cleaned up the corrupted cluster metadata
- Redeployed MySQL Router
Action items
- Monitor the health of the storage system more closely
- Improve the failover procedures for MySQL Router
- Investigate and fix the status page so it shows reliable information
Lessons
The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.
Sources
Related cases
IL-0408OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back emptyOpenRouter · 2026same root cause · shared Database
IL-0162LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hangLinode · 2025same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0489
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0489/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.