Cases1 case

1 to 1 of 1

1. IL-0252 Spotify users who were logged out could not log back in during a global outage of service discovery

   Spotify2022

   Symptoms

   At 18:12 UTC on 8 March 2022, reports began to surface of users being logged out of the different Spotify client apps. Anyone who was logged out of a Spotify app could not log back in, and the outage was global.

   High error rate

   Trigger

   Google Cloud Traffic Director, the xDS-based traffic control plane and discovery system that some Spotify services use, had an outage.

   Root cause

   Cloud provider outage

   Component

   Service discoveryHTTP and RPC client library

   Contributing factor

   Overlapping failures

   Detection

   Reports of users being logged out of the client apps started to surface at 18:12 UTC.

   Remediation

   • Rolled out configuration changes that moved the affected systems back to the DNS-based service discovery

   Action items

   • Work with Google Cloud to understand how the Traffic Director issues led to a large outage for Spotify users
   • Add monitoring and alerting that catches similar service discovery problems earlier
   • Add safety nets for monitoring, automatic error detection and self-recovery

   View case

By root cause1