IL-0514 Python Software Foundation · 2022-07-28kubernetesdatabase

PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down

PyPI runs on a Kubernetes cluster, with Consul and Vault clusters inside it as the management layer that deploys and secures PyPI's application servers. On 28 July 2022, from 17:24 UTC, the cloud provider automatically replaced several cluster instances because of an unknown failure, and all of them were healthy members of the cluster again by 18:20 UTC. The replacements moved many pods, and Consul and Vault restarted on the new nodes. Consul could not elect a leader and the new Vault containers stayed sealed. Without these services new PyPI containers failed, pods rescheduled from the replaced nodes could not start, and at 18:46 UTC PyPI became unresponsive.

The responding Infrastructure Team member worked first on Consul. Consul recovered only after the pod stuck in "Terminating" was forcefully removed and allowed to be recreated; the responder then unsealed the new Vault containers, and PyPI's backends were back at 18:53 UTC.

Severity: –
Scope: PyPI backends
Users affected: –
Region: –
Time to detect: –
Time to resolve: 7m

Symptoms

At 18:23 UTC on 28 July 2022, PSF Infrastructure got an alert that at least one Vault container in PyPI's Kubernetes cluster was down and needed unsealing. At 18:46 UTC PyPI's backends became unresponsive, and they stayed down until 18:53 UTC.

The responder found that every Vault container in the cluster was unavailable and that the Consul containers could not establish a new leader. One Consul pod was stuck in the "Terminating" state, and Consul's server discovery kept waiting for that extra server to take part in the election, which held up the recovery of Consul.

While Consul and Vault were offline, new containers started to serve PyPI's applications failed, so containers rescheduled after the node replacements could not start. As Kubernetes rescheduled more and more containers to reach the specified instance count, PyPI became unresponsive.

Symptom Service outage Pods stuck in Terminating

Root cause

Root cause Cloud provider outage

Component Coordination and key-value store Secrets store

Contributing factor Automation amplification

Trigger: From 17:24 UTC the cloud provider automatically replaced several instances of PyPI's Kubernetes cluster because of an unknown failure. All replacements were healthy by 18:20 UTC, but the event reshuffled many pods and restarted Consul and Vault on new nodes.

Detection

An alert at 18:23 UTC told PSF Infrastructure that at least one Vault container was down and needed unsealing.

Remediation

Forcefully removed the Consul pod stuck in Terminating so it was recreated and Consul could elect a leader
Unsealed the new Vault containers

Action items

–

Lessons

–

Sources

https://status.python.org/incidents/wcwq0pp25ffb · official-status · archived copy

Related cases

IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018
same root cause · shared Coordination and key-value store
IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degradedPagerDuty · 2013 · sev1
same root cause · shared Coordination and key-value store

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store, Secrets store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0514
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0514/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.