Cases1 case

1 to 1 of 1

1. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down

   Python Software Foundation2022

   Symptoms

   At 18:23 UTC on 28 July 2022, PSF Infrastructure got an alert that at least one Vault container in PyPI's Kubernetes cluster was down and needed unsealing. At 18:46 UTC PyPI's backends became unresponsive, and they stayed down until 18:53 UTC.

   Service outagePods stuck in Terminating

   Trigger

   From 17:24 UTC the cloud provider automatically replaced several instances of PyPI's Kubernetes cluster because of an unknown failure. All replacements were healthy by 18:20 UTC, but the event reshuffled many pods and restarted Consul and Vault on new nodes.

   Root cause

   Cloud provider outage

   Component

   Coordination and key-value storeSecrets store

   Contributing factor

   Automation amplification

   Detection

   An alert at 18:23 UTC told PSF Infrastructure that at least one Vault container was down and needed unsealing.

   Remediation

   • Forcefully removed the Consul pod stuck in Terminating so it was recreated and Consul could elect a leader
   • Unsealed the new Vault containers

   View case

By root cause1