Cases2 cases

1 to 2 of 2

1. IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty

   OpenRouter2026

   Symptoms

   At 5:27 AM UTC on 17 February 2026 part of OpenRouter's users started getting 500 errors on all API endpoints, and from 5:40 AM many requests failed with 401 "User not found" errors although their API keys were valid. The chatroom was degraded at the same time. The same pattern came back at 7:36 AM UTC on 19 February.

   High error rateService outageDatabase overload

   Trigger

   The third-party caching layer in front of API key lookups dropped all of its connections to OpenRouter's database and began returning errors.

   Root cause

   Dependency failure

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cacheOverlapping failures

   Detection

   An internal alert fired one minute after impact began on 17 February, at 5:28 AM UTC, and two minutes after it began on 19 February, at 7:38 AM UTC, and the response team came online to investigate.

   Remediation

   • Deployed WAF rules to block the denial-of-service attack running at the same time on 17 February

   Action items

   • Add circuit breakers that detect caching layer failures, limit their blast radius and repopulate a cold cache in a controlled way
   • Return 503 instead of 401 when an authorization lookup cannot complete because of an infrastructure problem
   • The caching provider fixes the failure mode that caused both incidents
   • Roll out a fallback caching mechanism so OpenRouter survives even extended downtime of the caching layer

   Lessons

   Answering an infrastructure failure with an authentication error made customers debug API keys that were fine, so error codes have to say what actually failed. Attributing the first outage to a concurrent denial-of-service attack, because the caching dependency had always been reliable, left the real fault in place until it recurred two days later. OpenRouter holds that redundancy against failures in its dependencies is its own responsibility, not the provider's.

   View case
2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down

   Magic Pages2025

   Symptoms

   On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

   Service outage

   Trigger

   One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

   Root cause

   Dependency failure

   Component

   DatabasePersistent volumes and stateful sets

   Contributing factor

   Harmful manual actionSingle point of failureMonitoring blind spot

   Remediation

   • Restarted the storage server to fix PVC provisioning
   • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
   • Cleaned up the corrupted cluster metadata
   • Redeployed MySQL Router

   Action items

   • Monitor the health of the storage system more closely
   • Improve the failover procedures for MySQL Router
   • Investigate and fix the status page so it shows reliable information

   Lessons

   The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

   View case

By root cause1