Incident Library
Applied filters Root causeCapacity exhaustionFactorNo headroomOrganisationDiscordClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
    Discord2023
    Symptoms

    On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

    Service outageDatabase overloadHigh latency
    Trigger

    During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm
    Detection

    An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

    Remediation
    • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
    • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
    • Restarted the two zone [b] nodes one at a time
    Action items
    • Double the size of the scylla-authentication cluster so each node takes half as many requests
    • Rebuild scylla-authentication-b-2 with new, non-degraded disks
    • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
    • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
    • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region
    Lessons

    Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Database
Contributing factor
Unverified assumptionIgnored early signalNo headroomOverlapping failuresRetry stormDependence on a warm cache
Organisation
Discord
  1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
    Discord2023