IL-0261 Discord · 2017-11-17gcpnetworking

Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distribution

An upgrade that Google rolled out to the Google Cloud Platform networking stack introduced a bug that caused short losses of network connectivity on Discord's VMs. Only the sessions cluster was affected, because of how heavily that cluster uses the network. Over a 24-hour period on 17 and 18 November the bug caused 16 separate loss of network events on sessions nodes, which gave two full outages of about 30 minutes each and fourteen smaller interruptions for parts of the user base.

Problems inside Discord made the network losses much worse. Under this kind of network instability, net_kernel, the Erlang VM component for internode communication, became overloaded because of how Discord's clusters and nodes talk to each other; the Erlang distribution port became wedged and memory grew from queued messages. Discord's internal service discovery component degraded from performance issues in its implementation, and the HTTP library used in Discord's Erlang stack became unstable under these conditions, which slowed recovery. On 17 November, the forceful termination of sessions-1-13 at 14:13 PST triggered a cascading failure and engineers restarted several core services; the service was back at 15:07. On 18 November another networking incident cascaded again, and engineers disabled message sending globally, restarted every member of the sessions cluster and re-enabled sending at 12:31.

Discord escalated its Google support ticket to P1 and set up a voice bridge with Google engineering, but Google did not find a problem on its side for most of the incident. At 13:30 on 18 November Google began rolling back a software change made to its networking stack on Friday, and around 15:30 Google confirmed the rollback was complete; diagnosis and resolution on Google's side took over 24 hours. After the rollback Discord saw no more networking issues.

Severity: –
Scope: Discord service completely offline for two periods of about 30 minutes, plus fourteen minor interruptions for portions of the userbase, over 24 hours
Users affected: all Discord users during the two full outages; millions of connected clients had to reconnect during the restarts
Region: –
Time to detect: 13m
Time to resolve: 24h 52m

Symptoms

On 17 and 18 November 2017 nodes of Discord's 'sessions' cluster on Google Cloud Platform lost network connectivity for short periods, 16 times in all. Engineers saw a service impact, a node whose service had died and restarted, and pages for low memory on other sessions nodes; twice the failure cascaded and Discord was completely offline for about 30 minutes.

On sessions-1-16 the network event triggered an application bug that killed the service, which restarted automatically and lost its clients. On sessions-1-9 a similar event briefly cut the node off from members of the guilds cluster; the service did not restart, the node was left in an invalid state, and the on-call engineer was paged for low memory. Engineers found that the "Erlang distribution port", which the nodes use to talk to each other, appeared to have failed, a state seen in earlier outages. The network instability overloaded net_kernel, the Erlang VM component that handles communication between nodes, the distribution port became "wedged", and queued messages made the process's memory grow without limit.

On 17 November engineers forcefully terminated sessions-1-9, sessions-1-14 and sessions-1-13 one after another. The first two terminations had no visible side effects, but the one at 14:13 PST triggered a cascading service failure, and engineers restarted several core services, which forced millions of connected clients to reconnect over about 20 minutes. Further sessions nodes misbehaved in the afternoon and evening. On 18 November at 11:38 a networking incident hit sessions-1-11 and the system cascaded towards failure again; a node of the guilds cluster also failed, and recovery after the restart was abnormally slow because of what engineers believed was another application bug.

At 12:11 on 17 November an engineer noted that the behaviour resembled a network split, which had been seen before on Google's compute engine network. Discord opened a P2 ticket with Google support and raised it to P1, but Google at first had no indication of a problem on its side and still did not believe there was one that evening, while Discord was certain it was a Google networking issue. Discord then noticed that no machine had failed again after a forceful termination and reboot, and began terminating and rebooting the whole sessions cluster.

Symptom Network unreachable Memory pressure Service outage

Root cause

Root cause Cloud provider outage

Component Service discovery

Contributing factor Limited debug access Harmful manual action

Trigger: A software change that Google rolled out to the Google Cloud Platform networking stack on Friday introduced a bug that made VMs in Discord's sessions cluster lose network connectivity for short periods.

Detection

At 11:52 PST on 17 November engineers noticed a service impact and escalated it in the incident chat. Later affected nodes showed up as pages to the on-call engineer for low memory on sessions nodes, and on 18 November as a page for a networking incident on sessions-1-11 together with issue reports that the customer experience team escalated from support channels.

Remediation

Forcefully terminated sessions nodes whose Erlang distribution port had failed
Escalated the Google support ticket to P1 and notified the Technical Account Manager
Restarted several core Discord services on 17 November, making clients reconnect over about 20 minutes
Began terminating and rebooting the entire sessions cluster after seeing that rebooted machines did not fail again
Disabled message sending globally on 18 November to help the service recover
Restarted all members of the sessions cluster
Re-enabled message sending after the initial all clear
Google rolled back the software change made to its networking stack

Action items

Continue the project to improve how Discord's Erlang clusters handle node failure
Expand how upgrades to these services are load tested so the rollout can be completed safely
Replace the HTTP library used by Discord's Elixir/Erlang services
Fix the instability seen in Erlang's net_kernel component
Replace the pieces of the service discovery component that degraded with more performant implementations
Work with Google on the issues Discord met while working with Google on this incident
Expand the internal procedure for communicating with Google so the progress of P1 issues is tracked

Lessons

Discord concluded that performance and stability issues in its own system, in Erlang's net_kernel, the service discovery component and the HTTP library, were what turned short network losses on part of the fleet from isolated issues for some users into two full system outages.

Sources

https://discordstatus.com/incidents/mpzkz7m4fzpt · official-status · archived copy

Related cases

IL-0252 Spotify users who were logged out could not log back in during a global outage of service discoverySpotify · 2022
same root cause · shared Service discovery

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0261
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0261/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.