Incident Library
Applied filters ComponentSecrets storeSymptomService outageFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022
    Symptoms

    At 18:23 UTC on 28 July 2022, PSF Infrastructure got an alert that at least one Vault container in PyPI's Kubernetes cluster was down and needed unsealing. At 18:46 UTC PyPI's backends became unresponsive, and they stayed down until 18:53 UTC.

    Service outagePods stuck in Terminating
    Trigger

    From 17:24 UTC the cloud provider automatically replaced several instances of PyPI's Kubernetes cluster because of an unknown failure. All replacements were healthy by 18:20 UTC, but the event reshuffled many pods and restarted Consul and Vault on new nodes.

    Root cause
    Cloud provider outage
    Component
    Coordination and key-value storeSecrets store
    Contributing factor
    Automation amplification
    Detection

    An alert at 18:23 UTC told PSF Infrastructure that at least one Vault container was down and needed unsealing.

    Remediation
    • Forcefully removed the Consul pod stuck in Terminating so it was recreated and Consul could elect a leader
    • Unsealed the new Vault containers
    View case
Check your cluster for this

This case involves Coordination and key-value store and Secrets store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Coordination and key-value storeSecrets store
Contributing factor
Automation amplification
Organisation
Python Software Foundation
  1. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022