IL-0388 Discord · 2017-10-13databasegcp
A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds
A Google Cloud Platform automatic migration made the primary node of a highly available Redis cluster used by Discord's API services drop offline. The cluster failed over to its secondary, but the failover triggered a bug, already known and scheduled to be fixed within weeks, in how the API instances handle Redis failover. Engineers escalated a P1 ticket with GCP support at 14:02 PDT and ran a rolling restart of all API instances at 14:11. Latency stayed high, and engineers found a misconfigured edge caching rule for an expensive API route that made clients call it aggressively. The requests overloaded a non-critical Cassandra cluster. Engineers corrected the rule, disabled the endpoint on the edge network, recovered Cassandra and restarted the API instances again, and the API was back to normal at 14:57.
At 15:41 new anomalies appeared: a node of the "guilds" cluster made part of the guilds appear offline, and nodes of the "sessions" cluster misbehaved. Engineers removed the guilds node from the service hashring, restarted it, and at 16:07 decided on a full restart of the service to reset the invalid state, forcing all users to reconnect. During the reboot the "push" component failed, and engineers found that, through human error, an API setting that lets users reconnect quickly and safely was set wrong. After it was corrected, service recovered slowly from 16:33 and was fully restored at 16:52, with millions of clients reconnected over about 20 minutes.
Discord could not say exactly why the partial API outage spread into a full system failure. The first failure made nodes of other services behave incorrectly until they ran out of memory and failed. In a later update, after alerts added for this postmortem caught similar behaviour on a node, Discord found possible flaws in a library that service uses and began rolling out an update.
- Severity
- –
- Scope
- API latency and errors, then guilds appearing offline and connection failures, ending in a full restart of Discord that reconnected all clients
- Users affected
- Discord users; millions of clients reconnected over about 20 minutes
- Region
- –
- Time to detect
- 0m
- Time to resolve
- 2h 51m
Symptoms
At about 14:01 PDT on 13 October 2017, Discord engineers were alerted to anomalies by real-time metrics and monitoring. A Redis instance that was the primary of a highly available cluster used by the API services had dropped offline after Google Cloud Platform migrated it automatically, and some API instances did not handle the failover to the secondary node correctly.
The Redis cluster itself had failed over properly to the secondary node, and the old primary was completely unavailable. After a rolling restart of all API instances, the API endpoints still showed abnormal latency. A misconfigured edge caching rule for an expensive API route made clients call that route aggressively, and the volume of requests made a non-critical Cassandra cluster fall over under load.
At 15:41 PDT, after the API had returned to normal at 14:57, monitoring showed another anomaly, and user reports said some servers appeared offline. A node of the "guilds" cluster was misbehaving, so part of Discord's guilds appeared offline, and then nodes of the "sessions" cluster misbehaved too. During the full restart that followed, a component called "push" also failed. Nodes of other services had run out of memory and failed, which spread the failure through the real-time system.
Root cause
Root cause Software defect
- Trigger
- Google Cloud Platform automatically migrated the Redis instance that was the primary of a highly available cluster used by the API services, and the node incorrectly dropped offline.
Detection
Real-time metrics and monitoring alerted engineers at 14:01 PDT. The second phase was seen at 15:41 as another anomaly in monitoring, and end user reports of servers appearing offline were escalated at 15:42.
Remediation
- Ran a rolling restart of all API instances to clear the Redis connection issues
- Corrected the misconfigured edge caching rule for the expensive API route
- Disabled the problem API endpoint on the edge network
- Recovered the Cassandra cluster
- Ran a second rolling restart of all API instances for the latency left by the Cassandra issues
- Removed the misbehaving guilds node from the service hashring
- Forcefully restarted the misbehaving guilds node
- Restarted the whole service, forcing all users to reconnect
- Blocked traffic that relied on the failing push component to reduce its load
- Corrected the API setting that lets users reconnect quickly and safely
Action items
- Fix the known issues with Redis failover within the coming weeks
- Change the cached route so this type of error does not degrade other services
- Monitor the signals that would have warned of the cascading failure
- Improve the reliability and failure handling of the affected services
- Roll out an update of the library with the suspected flaws
Lessons
–
Sources
- https://status.discordapp.com/incidents/qk9cdgnqnhcn · official-status · archived copy
Related cases
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared In-memory cache, Database
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0226Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errorsMicrosoft Azure · 2025same root cause · shared CDN and edge network
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0376A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databasesUpstash · 2025same root cause · shared In-memory cache
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0216Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfwWikimedia Foundation · 2023same root cause · shared In-memory cache
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0295Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022same root cause · shared In-memory cache
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0213Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021same root cause · shared In-memory cache
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0279One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minuteGrab · 2019same root cause · shared In-memory cache
IL-0338Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requestsGoogle Cloud · 2018same root cause · shared CDN and edge network
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0388
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0388/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.