Incident Library
Applied filters Root causeDependency failureFactorSingle point of failureOrganisationMagic PagesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
    Symptoms

    On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

    Service outage
    Trigger

    One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

    Root cause
    Dependency failure
    Component
    DatabasePersistent volumes and stateful sets
    Contributing factor
    Harmful manual actionSingle point of failureMonitoring blind spot
    Remediation
    • Restarted the storage server to fix PVC provisioning
    • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
    • Cleaned up the corrupted cluster metadata
    • Redeployed MySQL Router
    Action items
    • Monitor the health of the storage system more closely
    • Improve the failover procedures for MySQL Router
    • Investigate and fix the status page so it shows reliable information
    Lessons

    The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

    View case
Check your cluster for this

This case involves Database and Persistent volumes and stateful sets. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
DatabasePersistent volumes and stateful sets
Contributing factor
Harmful manual actionMonitoring blind spotSingle point of failure
Organisation
Magic Pages
  1. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025