IL-0513 Python Software Foundation · 2022-10-24kubernetesawsnetworking
PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
PyPI and several ancillary services run as Deployments in a Kubernetes cluster and are exposed through Ingress with the AWS Elastic Load Balancer (ELB) integration. The ELB's TLS certificate is managed in Amazon's Certificate Manager (ACM), and the Ingress had been set up with an existing ACM certificate. After the previous rolling upgrade, PyPI's Ingress needed an additional hostname, so PyPI administrators created a new ACM certificate with that hostname and updated the Ingress-managed ELB to use it by hand in the AWS console, while the Ingress configuration still named the old certificate.
On 24 October 2022 a rolling restart of all nodes brought up new Kubernetes API servers. They re-validated the Ingress configurations, could not find the old ACM certificate, and disabled the HTTPS listener for the Ingress serving PyPI. Once the cause was found, the administrators pointed the Ingress configuration at the new ACM certificate, Kubernetes restored the HTTPS listener on the ELB, and access came back at 15:03 UTC.
- Severity
- –
- Scope
- PyPI backends over HTTPS: CDN page fetches, uploads to uploads.pypi.org and the legacy file redirect service
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- 20m
Symptoms
From 14:43 to 15:03 UTC on 24 October 2022, during regular maintenance of the Kubernetes cluster that hosts PyPI, PyPI's backends could not be reached over HTTPS. The CDN could not fetch pages, uploads to uploads.pypi.org were impossible, and other services such as the legacy file redirect service were interrupted.
The HTTPS listener of the AWS Elastic Load Balancer that the Kubernetes Ingress manages had been disabled for the Ingress serving PyPI and its associated services. The status page listed pypi.org backends, files.pythonhosted.org redirects and the redirects backends as affected.
Symptom Service outage
Root cause
Root cause Configuration change
- Trigger
- Regular maintenance did a rolling restart of every node in the cluster to roll out upgrades and new configuration; as the new Kubernetes API server hosts came up, Kubernetes validated and refreshed the Ingress configurations.
Detection
–
Remediation
- Pointed the Ingress configuration at the new ACM TLS certificate, and Kubernetes restored the HTTPS listener on the ELB
Action items
- Investigate managing the hostnames on the ACM TLS certificates for PyPI's Ingress through Kubernetes resources instead of the AWS console
Lessons
When every resource is managed through the Kubernetes API, drift between the desired state and reality is less likely to build up and then surface during later maintenance.
Sources
- https://status.python.org/incidents/77qzc7qsb5wb · official-status · archived copy
Related cases
IL-0177WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE Plus · 2022same root cause · shared Ingress controller, Cloud load balancer
IL-0111Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021same root cause · shared Ingress controller, Cloud load balancer
IL-0346Render's routing service returned intermittent 404s for web services and static sites in OregonRender · 2025same root cause · shared Cloud load balancer
IL-0543Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutesTwingate · 2025same root cause · shared Cloud load balancer
IL-0192A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Cloud load balancer
IL-0473Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate themJenkins project · 2023same root cause · shared Cloud load balancer
IL-0208A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022same root cause · shared Cloud load balancer
IL-0442Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spikedLIFULL · 2021same root cause · shared Ingress controller
IL-0105Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021same root cause · shared Cloud load balancer
IL-0365A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020same root cause · shared Cloud load balancer
IL-0052A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019same root cause · shared Cloud load balancer
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0513
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0513/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.