Cases2 cases

1 to 2 of 2

1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
2. IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds

   Discord2017

   Symptoms

   At about 14:01 PDT on 13 October 2017, Discord engineers were alerted to anomalies by real-time metrics and monitoring. A Redis instance that was the primary of a highly available cluster used by the API services had dropped offline after Google Cloud Platform migrated it automatically, and some API instances did not handle the failover to the secondary node correctly.

   Failover failureHigh latencyService outage

   Trigger

   Google Cloud Platform automatically migrated the Redis instance that was the primary of a highly available cluster used by the API services, and the node incorrectly dropped offline.

   Root cause

   Software defect

   Component

   In-memory cacheDatabaseCDN and edge network

   Contributing factor

   Unpatched known bugOverlapping failuresMonitoring blind spot

   Detection

   Real-time metrics and monitoring alerted engineers at 14:01 PDT. The second phase was seen at 15:41 as another anomaly in monitoring, and end user reports of servers appearing offline were escalated at 15:42.

   Remediation

   • Ran a rolling restart of all API instances to clear the Redis connection issues
   • Corrected the misconfigured edge caching rule for the expensive API route
   • Disabled the problem API endpoint on the edge network
   • Recovered the Cassandra cluster
   • Ran a second rolling restart of all API instances for the latency left by the Cassandra issues
   • Removed the misbehaving guilds node from the service hashring
   • Forcefully restarted the misbehaving guilds node
   • Restarted the whole service, forcing all users to reconnect
   • Blocked traffic that relied on the failing push component to reduce its load
   • Corrected the API setting that lets users reconnect quickly and safely

   Action items

   • Fix the known issues with Redis failover within the coming weeks
   • Change the cached route so this type of error does not degrade other services
   • Monitor the signals that would have warned of the cascading failure
   • Improve the reliability and failure handling of the affected services
   • Roll out an update of the library with the suspected flaws

   View case

By root cause2