Incident Library

IL-0408 OpenRouter · 2026-02-17ai-infradatabase

OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty

OpenRouter checks API keys against its database through an external caching layer from a third-party provider; normally the cache answers almost every authentication check and only a small fraction of requests reach the database. In both incidents the caching layer dropped all of its connections to the database and started returning errors, which failed about 20% of requests with 500. Within minutes the cache recovered and reconnected, but all cached entries had been invalidated, so every incoming request needed a fresh database lookup. The database could not handle the sudden spike in lookups, and the resulting timeouts reached users as 401 "User not found" errors.

On 17 February an internal alert fired at 5:28 AM, a minute after impact began. A concurrent denial-of-service attack was taken for the cause and blocked with WAF rules at 5:50, and service came back when the cache had fully repopulated at 6:05, 38 minutes after the start. The caching provider began investigating after the first incident, but diagnosing and fixing the failure took time. On 19 February the cache failed the same way at 7:36 AM without an attack, the alert fired at 7:38, and service was restored when the cache had repopulated at 8:11, after 35 minutes. OpenRouter then deployed targeted protections within hours. The changes it has deployed are circuit breakers that detect caching layer failures and refill a cold cache in a controlled way instead of letting every request fall through to the database at once, and a 503 instead of a 401 when an authorization lookup cannot complete. The caching provider deployed fixes for the failure mode behind both incidents.

Severity
–
Scope
All OpenRouter API endpoints and the chatroom: on 17 February about 20% of API requests failed, then 80-90%; on 19 February partial failures, then near-total downtime
Users affected
A portion of OpenRouter users
Region
–
Time to detect
1m
Time to resolve
38m

Symptoms

At 5:27 AM UTC on 17 February 2026 part of OpenRouter's users started getting 500 errors on all API endpoints, and from 5:40 AM many requests failed with 401 "User not found" errors although their API keys were valid. The chatroom was degraded at the same time. The same pattern came back at 7:36 AM UTC on 19 February.

On 17 February about 20% of API requests failed between 5:27 and 5:40 AM UTC, and 80-90% failed between 5:40 and 6:05 AM. On 19 February failures were partial between 7:36 and 7:42 AM and close to total between 7:42 and 8:11 AM. Both times the caching layer had logged occasional errors with no user-facing impact beforehand, from 5:20 AM and from 7:16 AM.

The 401 errors sent some customers into debugging their own API key configurations when nothing on their side was wrong. They were database timeouts during API key lookups, returned to users as an authentication failure.

On 17 February a denial-of-service attack was ramping up at the moment the caching layer failed. Because such attacks can cause cascading problems across the stack, and because the caching dependency had historically been extremely reliable, OpenRouter first attributed the outage to the attack, deployed WAF rules against it at 5:50 AM and put its effort into hardening against that attack vector. Only when the same cache failure recurred on 19 February without an attack did the real root cause become clear.

Symptom High error rateService outageDatabase overload

Root cause

Root cause Dependency failure

Component In-memory cacheDatabase
Contributing factor Dependence on a warm cacheOverlapping failures
Trigger
The third-party caching layer in front of API key lookups dropped all of its connections to OpenRouter's database and began returning errors.

Detection

An internal alert fired one minute after impact began on 17 February, at 5:28 AM UTC, and two minutes after it began on 19 February, at 7:38 AM UTC, and the response team came online to investigate.

Remediation

Action items

Lessons

Answering an infrastructure failure with an authentication error made customers debug API keys that were fine, so error codes have to say what actually failed. Attributing the first outage to a concurrent denial-of-service attack, because the caching dependency had always been reliable, left the real fault in place until it recurred two days later. OpenRouter holds that redundancy against failures in its dependencies is its own responsibility, not the provider's.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.