Cases2 cases

1 to 2 of 2

1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
2. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed

   Mezmo2019

   Symptoms

   On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

   Service outageOOMKilledDatabase overloadThroughput degradation

   Trigger

   A traffic spike that the tool tracking batches of logs was not sized to handle.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cache

   Detection

   Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

   Remediation

   • Resized specific parts of the cluster to handle a much higher load

   Action items

   • Retool the batch tracking system to reduce its direct impact on the Redis clusters
   • Add observability instrumentation around the batch tracking feature set
   • Improve monitoring for Redis
   • Remove the bottlenecks in the pipeline that were involved in this incident

   Lessons

   Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

   View case

By root cause1