Incident Library
Applied filters ComponentIngress controllerFactorChange with a delayed effectOrganisationPython Software FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
    Symptoms

    From 14:43 to 15:03 UTC on 24 October 2022, during regular maintenance of the Kubernetes cluster that hosts PyPI, PyPI's backends could not be reached over HTTPS. The CDN could not fetch pages, uploads to uploads.pypi.org were impossible, and other services such as the legacy file redirect service were interrupted.

    Service outage
    Trigger

    Regular maintenance did a rolling restart of every node in the cluster to roll out upgrades and new configuration; as the new Kubernetes API server hosts came up, Kubernetes validated and refreshed the Ingress configurations.

    Root cause
    Configuration change
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Configuration not in codeChange with a delayed effect
    Remediation
    • Pointed the Ingress configuration at the new ACM TLS certificate, and Kubernetes restored the HTTPS listener on the ELB
    Action items
    • Investigate managing the hostnames on the ACM TLS certificates for PyPI's Ingress through Kubernetes resources instead of the AWS console
    Lessons

    When every resource is managed through the Kubernetes API, drift between the desired state and reality is less likely to build up and then surface during later maintenance.

    View case
Check your cluster for this

This case involves Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Cloud load balancerIngress controller
Contributing factor
Configuration not in codeChange with a delayed effect
Organisation
Python Software Foundation
  1. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022