IL-0252 Spotify · 2022-03-08gcpnetworking

Spotify users who were logged out could not log back in during a global outage of service discovery

Spotify's backend microservices find each other through several service discovery systems: most use a DNS-based system, and some use Google Cloud Traffic Director, an xDS-based traffic control plane and discovery system. On 8 March 2022 Traffic Director had an outage, and together with a bug in a gRPC client library this broke login for Spotify users worldwide. Spotify learned of the login problems at 18:12 UTC, started remediation at 18:39 UTC by rolling out configuration changes that moved the affected systems back to the DNS-based service discovery, saw them recover gradually, and had the incident fully mitigated at 20:35 UTC.

Severity: –
Scope: Login to Spotify apps: logged-out users could not log back in
Users affected: –
Region: global
Time to detect: –
Time to resolve: –

Symptoms

At 18:12 UTC on 8 March 2022, reports began to surface of users being logged out of the different Spotify client apps. Anyone who was logged out of a Spotify app could not log back in, and the outage was global.

Symptom High error rate

Root cause

Root cause Cloud provider outage

Component Service discovery HTTP and RPC client library

Contributing factor Overlapping failures

Trigger: Google Cloud Traffic Director, the xDS-based traffic control plane and discovery system that some Spotify services use, had an outage.

Detection

Reports of users being logged out of the client apps started to surface at 18:12 UTC.

Remediation

Rolled out configuration changes that moved the affected systems back to the DNS-based service discovery

Action items

Work with Google Cloud to understand how the Traffic Director issues led to a large outage for Spotify users
Add monitoring and alerting that catches similar service discovery problems earlier
Add safety nets for monitoring, automatic error detection and self-recovery

Lessons

–

Sources

https://engineering.atspotify.com/2022/3/incident-report-spotify-outage-on-march-8 · blog · archived copy

Related cases

IL-0261 Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distributionDiscord · 2017
same root cause · shared Service discovery

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Service discovery, HTTP and RPC client library among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0252
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0252/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.