Cases2 cases

1 to 2 of 2

1. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener

   Python Software Foundation2022

   Symptoms

   From 14:43 to 15:03 UTC on 24 October 2022, during regular maintenance of the Kubernetes cluster that hosts PyPI, PyPI's backends could not be reached over HTTPS. The CDN could not fetch pages, uploads to uploads.pypi.org were impossible, and other services such as the legacy file redirect service were interrupted.

   Service outage

   Trigger

   Regular maintenance did a rolling restart of every node in the cluster to roll out upgrades and new configuration; as the new Kubernetes API server hosts came up, Kubernetes validated and refreshed the Ingress configurations.

   Root cause

   Configuration change

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Configuration not in codeChange with a delayed effect

   Remediation

   • Pointed the Ingress configuration at the new ACM TLS certificate, and Kubernetes restored the HTTPS listener on the ELB

   Action items

   • Investigate managing the hostnames on the ACM TLS certificates for PyPI's Ingress through Kubernetes resources instead of the AWS console

   Lessons

   When every resource is managed through the Kubernetes API, drift between the desired state and reality is less likely to build up and then surface during later maintenance.

   View case
2. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down

   Python Software Foundation2022

   Symptoms

   At 18:23 UTC on 28 July 2022, PSF Infrastructure got an alert that at least one Vault container in PyPI's Kubernetes cluster was down and needed unsealing. At 18:46 UTC PyPI's backends became unresponsive, and they stayed down until 18:53 UTC.

   Service outagePods stuck in Terminating

   Trigger

   From 17:24 UTC the cloud provider automatically replaced several instances of PyPI's Kubernetes cluster because of an unknown failure. All replacements were healthy by 18:20 UTC, but the event reshuffled many pods and restarted Consul and Vault on new nodes.

   Root cause

   Cloud provider outage

   Component

   Coordination and key-value storeSecrets store

   Contributing factor

   Automation amplification

   Detection

   An alert at 18:23 UTC told PSF Infrastructure that at least one Vault container was down and needed unsealing.

   Remediation

   • Forcefully removed the Consul pod stuck in Terminating so it was recreated and Consul could elect a leader
   • Unsealed the new Vault containers

   View case

By root cause2