Cases3 cases

1 to 3 of 3

1. IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty

   OpenRouter2026

   Symptoms

   At 5:27 AM UTC on 17 February 2026 part of OpenRouter's users started getting 500 errors on all API endpoints, and from 5:40 AM many requests failed with 401 "User not found" errors although their API keys were valid. The chatroom was degraded at the same time. The same pattern came back at 7:36 AM UTC on 19 February.

   High error rateService outageDatabase overload

   Trigger

   The third-party caching layer in front of API key lookups dropped all of its connections to OpenRouter's database and began returning errors.

   Root cause

   Dependency failure

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cacheOverlapping failures

   Detection

   An internal alert fired one minute after impact began on 17 February, at 5:28 AM UTC, and two minutes after it began on 19 February, at 7:38 AM UTC, and the response team came online to investigate.

   Remediation

   • Deployed WAF rules to block the denial-of-service attack running at the same time on 17 February

   Action items

   • Add circuit breakers that detect caching layer failures, limit their blast radius and repopulate a cold cache in a controlled way
   • Return 503 instead of 401 when an authorization lookup cannot complete because of an infrastructure problem
   • The caching provider fixes the failure mode that caused both incidents
   • Roll out a fallback caching mechanism so OpenRouter survives even extended downtime of the caching layer

   Lessons

   Answering an infrastructure failure with an authentication error made customers debug API keys that were fine, so error codes have to say what actually failed. Attributing the first outage to a concurrent denial-of-service attack, because the caching dependency had always been reliable, left the real fault in place until it recurred two days later. OpenRouter holds that redundancy against failures in its dependencies is its own responsibility, not the provider's.

   View case
2. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
3. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed

   Mezmo2019

   Symptoms

   On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

   Service outageOOMKilledDatabase overloadThroughput degradation

   Trigger

   A traffic spike that the tool tracking batches of logs was not sized to handle.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cache

   Detection

   Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

   Remediation

   • Resized specific parts of the cluster to handle a much higher load

   Action items

   • Retool the batch tracking system to reduce its direct impact on the Redis clusters
   • Add observability instrumentation around the batch tracking feature set
   • Improve monitoring for Redis
   • Remove the bottlenecks in the pipeline that were involved in this incident

   Lessons

   Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

   View case

By root cause3