Cases6 cases

1 to 5 of 6

1. IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty

   OpenRouter2026

   Symptoms

   At 5:27 AM UTC on 17 February 2026 part of OpenRouter's users started getting 500 errors on all API endpoints, and from 5:40 AM many requests failed with 401 "User not found" errors although their API keys were valid. The chatroom was degraded at the same time. The same pattern came back at 7:36 AM UTC on 19 February.

   High error rateService outageDatabase overload

   Trigger

   The third-party caching layer in front of API key lookups dropped all of its connections to OpenRouter's database and began returning errors.

   Root cause

   Dependency failure

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cacheOverlapping failures

   Detection

   An internal alert fired one minute after impact began on 17 February, at 5:28 AM UTC, and two minutes after it began on 19 February, at 7:38 AM UTC, and the response team came online to investigate.

   Remediation

   • Deployed WAF rules to block the denial-of-service attack running at the same time on 17 February

   Action items

   • Add circuit breakers that detect caching layer failures, limit their blast radius and repopulate a cold cache in a controlled way
   • Return 503 instead of 401 when an authorization lookup cannot complete because of an infrastructure problem
   • The caching provider fixes the failure mode that caused both incidents
   • Roll out a fallback caching mechanism so OpenRouter survives even extended downtime of the caching layer

   Lessons

   Answering an infrastructure failure with an authentication error made customers debug API keys that were fine, so error codes have to say what actually failed. Attributing the first outage to a concurrent denial-of-service attack, because the caching dependency had always been reliable, left the real fault in place until it recurred two days later. OpenRouter holds that redundancy against failures in its dependencies is its own responsibility, not the provider's.

   View case
2. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
3. IL-0567 Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play

   Devsisters2023

   Symptoms

   After further bug-fix deployments and reconnecting users on the launch day of the package sales ranking feature, a database alarm fired and the CPU of the Cookie Run: OvenBreak AWS Aurora MySQL database was at 100%. Database connections then failed, and players could not log in to or play the game.

   Database overloadService outage

   Trigger

   The package sales ranking feature was deployed, and further bug-fix deployments followed that reconnected users; each server update left more servers with empty caches that ran the ranking query against the database.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryDependence on a warm cacheUnrepresentative staging

   Detection

   A database alarm fired, and the CPU graph of the Aurora MySQL database showed 100%.

   Remediation

   • Put the game servers into maintenance
   • Added a switch for the package sales ranking feature and turned the feature off, which restored the service
   • Rewrote the ranking query to group products that share a sale start time and to drop the current-time condition
   • Moved the ranking query to a scheduler server that runs it periodically and stores one shared cached result
   • Turned the package sales ranking feature back on after both fixes

   Lessons

   A heavy query combined with per-server caches in a distributed server setup caused a problem nobody expected, so the points likely to cause trouble need to be found in advance. During an outage, restoring the service comes before finding the cause, and a switch that turns a feature off, built in ahead of time, is a safety device that helps run a stable service. The online ALTER itself went well with Percona Toolkit and was used again for later schema changes.

   View case
4. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
5. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed

   Mezmo2019

   Symptoms

   On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

   Service outageOOMKilledDatabase overloadThroughput degradation

   Trigger

   A traffic spike that the tool tracking batches of logs was not sized to handle.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cache

   Detection

   Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

   Remediation

   • Resized specific parts of the cluster to handle a much higher load

   Action items

   • Retool the batch tracking system to reduce its direct impact on the Redis clusters
   • Add observability instrumentation around the batch tracking feature set
   • Improve monitoring for Redis
   • Remove the bottlenecks in the pipeline that were involved in this incident

   Lessons

   Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

   View case

By root cause4