Incident Library
Applied filters Root causeDependency failureComponentCDN and edge networkFactorPartial rolloutClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs
    Python Software Foundation2026
    Symptoms

    For about two weeks in August 2026, some PyPI users got intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, and installations from PyPI failed. The first two reports of persistent 502s were opened in the support tracker on 17 August.

    High error rate
    Trigger

    A partial rollback during a canary deployment inside Fastly's network, whose canary cohort had included PyPI's traffic for a number of years, left the caching configuration of one Seattle point of presence reverted while the routing configuration in front of it was not.

    Root cause
    Dependency failure
    Component
    CDN and edge networkObject storage
    Contributing factor
    Partial rolloutOverlapping failuresSignal buried in noise
    Detection

    Users filed reports in the PyPI support tracker; one report isolated the 502s to a single Fastly cache node using the x-served-by headers of the failing responses.

    Remediation
    • Fastly's network operations removed the routing override that sent part of PyPI's traffic to the affected point of presence
    • Fixed the B2-to-S3 fallback for the case where B2 does not respond at all (infra#237)
    • Exempted suffix and multi-range requests from segmented caching (infra#241, infra#243)
    • Fastly patched the canary configuration bug and removed all PSF traffic from its canary cohort
    • Moved the segmented-caching exemption check after URL normalization (infra#245)
    Action items
    • Improve the logging configuration of the file hosting service (infra#238)
    • Reject HTTP methods that a file host should not receive (infra#239)
    • Follow a support ticket with Fastly about segmented-caching range handling, and revert PyPI's own handling code once Fastly fixes it
    • Return to Fastly's canary program once clearer controls and notifications exist around that traffic
    • Hire an infrastructure engineer, partly for PyPI, to catch conditions like this earlier
    Lessons

    None of the PyPI configuration bugs were new; they had been in the configuration for a while and came out only when real traffic on large files with range requests forced an investigation. The 5xx baseline was already noisy, so the start of the Fastly problem was easy to miss, and part of that noise turned out to be one of the bugs.

    View case
Check your cluster for this

This case involves CDN and edge network and Object storage. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
CDN and edge networkObject storage
Contributing factor
Signal buried in noiseOverlapping failuresPartial rollout
Organisation
Python Software Foundation
  1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs
    Python Software Foundation2026