Incident Library

IL-0515 Python Software Foundation · 2026-08-15networking

Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs

files.pythonhosted.org is a Fastly CDN service in front of three origins. Uploaded files are written to Amazon S3 as the durable copy and synced to Backblaze B2; on downloads Fastly tries B2 first and falls back to S3 when B2 does not answer or answers unexpectedly, and a third backend, Conveyor, serves everything that is not a package file. From 15 August 2026, a partial rollback in a Fastly canary deployment left one Seattle-area cache node with reverted caching configuration behind routing configuration that had not been reverted, and the routing layer returned 502 for traffic reaching that node. On 19 August Fastly's network operations removed a routing override that sent part of PyPI's traffic to the affected point of presence, Fastly saw recovery there on 20 August, and on 28 August Fastly patched the canary configuration bug and removed all PSF traffic, including PyPI, from its canary cohort.

While investigating, PyPI's infrastructure engineer found older bugs in PyPI's own Fastly configuration that produced the same symptom. The fallback to S3 ran only when B2 answered with an error status; when B2 did not answer at all, Fastly synthesized its own 503 and skipped the fallback, until infra#237 added that path on 18 August. Segmented caching, used so a Range request does not pull a whole gigabyte-sized wheel into cache, cannot answer a suffix range such as bytes=-1024 or a range that starts past the end and returns a synthetic 501, and some installers read wheel metadata with suffix ranges; infra#241 on 21 August and infra#243 on 24 August exempted those requests, the second after two broken parallel downloaders generated 41,315 more errors of that class in a single day. infra#245 on 28 August moved an exemption check after URL normalization: a .metadata request with a query string had kept segmented caching on, got a 501 from the archive backend, and that response was cached and served to every later request for the same file.

After infra#241, the 5xx count fell to two to three orders of magnitude below the pre-incident baseline, so part of what had been treated as background noise had been this bug. Downloads were fully working again from 28 August.

Severity
–
Scope
Downloads from files.pythonhosted.org and therefore installations from PyPI
Users affected
some PyPI users
Region
Seattle-area Fastly point of presence for the Fastly fault
Time to detect
2d
Time to resolve
13d

Symptoms

For about two weeks in August 2026, some PyPI users got intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, and installations from PyPI failed. The first two reports of persistent 502s were opened in the support tracker on 17 August.

A report on 18 August reproduced the problem in detail: 88 recorded 502s over six hours across 32 unrelated packages, including the small .whl.metadata range requests installers use to read PEP 658 metadata. A report on 19 August traced the failures to one Fastly cache node, cache-pae2080020, which had returned 502 for every request routed to it for over 19 hours; the x-served-by headers on the failing responses confirmed it.

On the PyPI side the errors were hard to see. Datadog Logs showed nothing of consequence for the PyPI Files errors at first, and the 5xx baseline was already thousands to tens of thousands of responses, so the increase from 15 August was easy to miss. Fastly's real-time analytics showed B2 errors climbing from 15 August while the S3 archive backend stayed flat at zero, because the fallback was not firing. On 21 August a single client sending logically invalid ranges pushed the count close to a million, and some requests got 501 responses that looked like 502s from outside.

Symptom High error rate

Root cause

Root cause Dependency failure

Component CDN and edge networkObject storage
Contributing factor Partial rolloutOverlapping failuresSignal buried in noise
Trigger
A partial rollback during a canary deployment inside Fastly's network, whose canary cohort had included PyPI's traffic for a number of years, left the caching configuration of one Seattle point of presence reverted while the routing configuration in front of it was not.

Detection

Users filed reports in the PyPI support tracker; one report isolated the 502s to a single Fastly cache node using the x-served-by headers of the failing responses.

Remediation

Action items

Lessons

None of the PyPI configuration bugs were new; they had been in the configuration for a while and came out only when real traffic on large files with range requests forced an investigation. The 5xx baseline was already noisy, so the start of the Fastly problem was easy to miss, and part of that noise turned out to be one of the bugs.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.