IL-0270 Discord · 2023-11-06database
Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
Discord's scylla-authentication ScyllaDB cluster holds user session data and sits behind the Rust authentication service that validates every API request. Like Discord's other Scylla clusters, it replicates data across three zones of the cloud provider and uses quorum reads so that it can lose one zone, and each node mirrors fast NVMe Local SSDs, which are cleared on reboot, with a slower persistent disk in a RAID array. Syncing that RAID takes about 40 minutes at instance startup before Scylla can start. From 3 November the persistence infrastructure team upgraded the operating system of the Scylla clusters one zone at a time: detach the persistent disks, terminate the instances, start new instances with the upgraded OS, rebuild the RAID, bring Scylla back and wait a set time before the next zone. While a zone is out, the cluster has a third less compute and no room for another failure, and the detached capacity cannot return for at least 45 minutes. More than a hundred nodes had been upgraded this way. Because scylla-authentication had not had a full zonal restart recently, the team first took zone [b] offline for ten minutes as a test, saw no degradation, and started the upgrade at 9:16 AM.
Zone [b] finished at 10:42 AM. One of its two nodes, scylla-authentication-b-2, had rebuilt its RAID much more slowly than the other because one of its NVMe Local SSDs was degraded, which hurt that node's disk I/O. The operator checked the cluster and advanced to the next zone, and at 10:52 AM the script drained zone [c]. The cluster was left with one degraded node and one healthy node in zone [b], both with fairly empty row caches, zone [c] offline and two healthy nodes in zone [d]. Discord had found that for some clusters the ScyllaDB row cache hit rate must reach a certain level before restarting nodes is safe, and the procedure had not checked the hit rate of the restarted zone [b] nodes; the earlier ten-minute test had given false confidence. Discord's write-up names the cluster as underprovisioned. It overloaded, authentication lookups timed out, and application retries made the overload worse.
Discord shed load by turning off typing events and read markers at 11:05 AM, which restored authentication for about half of users. Restarting nodes to clear abandoned queries risked breaking quorum completely, so at 11:35 AM, with the zone [c] RAID rebuild five minutes from done, the team chose to wait. The two zone [c] nodes came back at 11:41 AM, the two zone [b] nodes were restarted one at a time, and by 11:43 AM user activity was back to normal. Right after the incident the cluster was doubled in size, and after some analysis scylla-authentication-b-2 was rebuilt on new disks. The outage affected availability only; the authentication service returned no incorrect results, and no outside actor or unauthorized access was involved.
- Severity
- –
- Scope
- Discord authentication: users could not connect, log in, read or send messages; availability fell to about 25%, then about 50% of users could use Discord until 11:43 AM PST
- Users affected
- Discord users; about 50% could use the service from 11:05 AM until recovery
- Region
- –
- Time to detect
- 4m
- Time to resolve
- 51m
Symptoms
On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.
The first alerts fired at 10:56 AM, but an incident call was already running. A graph of time spent by downstream service showed that the Rust authentication service, which checks that every API request comes from an authenticated user, was taking far longer than normal on each request, and the authentication alerts fired moments later. The scylla-authentication cluster was saturated with queries and showed high latency; all four remaining nodes were overwhelmed, with load at 100% on some of their shards. Scylla's logs reported errors connecting to the two zone [b] nodes, scylla-authentication-b-1 and scylla-authentication-b-2.
The application retries requests that get server errors from the authentication endpoints, so once the cluster began timing out, the retries added a retry storm on top of the load. Turning off typing events and read markers at 11:05 AM brought authentication back for about 50% of users, and at 11:10 AM still only about half of users could use Discord.
The timing pointed to the authentication upgrade right away, but the script had already detached zone [c]'s data disks, so bringing that zone back meant about 45 minutes of instance provisioning and RAID rebuild. Restarting nodes to clear long-running queries abandoned by clients was considered, but with zone [c] offline a restart would break quorum, and restarting the wrong node could take the service down completely. A misleading error message in the authentication service sent the investigation on a detour. Earlier that morning, while zone [b] was being upgraded, scylla-authentication-b-2 had rebuilt its RAID array much more slowly than the other node, which the team had treated as a curiosity.
Root cause
Root cause Capacity exhaustion
- Trigger
- During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.
Detection
An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.
Remediation
- Turned off sending typing events and marking messages as read to shed load from the authentication cluster
- Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
- Restarted the two zone [b] nodes one at a time
Action items
- Double the size of the scylla-authentication cluster so each node takes half as many requests
- Rebuild scylla-authentication-b-2 with new, non-degraded disks
- Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
- Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
- Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region
Lessons
Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.
Sources
Related cases
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0270
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0270/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.