Cases2 cases

1 to 2 of 2

1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

   Google Cloud2026

   Symptoms

   From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

   High latencyHigh error rateLoad imbalance

   Trigger

   A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

   Root cause

   Configuration change

   Component

   Object storageDatabase

   Contributing factor

   No headroomOverlapping failures

   Detection

   Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

   Remediation

   • Drained traffic from the most overloaded storage backend nodes
   • Deployed a configuration change that disabled the traffic sorting optimization
   • Rolled back a recent update to the metadata database service
   • Redirected traffic and drained affected storage components again during the recurrence
   • Fully reverted the load balancing configuration change, restoring capacity-aware routing
   • Returned the drained nodes to service once resource use was back to normal

   Action items

   • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
   • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
   • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes

   View case
2. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs

   Python Software Foundation2026

   Symptoms

   For about two weeks in August 2026, some PyPI users got intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, and installations from PyPI failed. The first two reports of persistent 502s were opened in the support tracker on 17 August.

   High error rate

   Trigger

   A partial rollback during a canary deployment inside Fastly's network, whose canary cohort had included PyPI's traffic for a number of years, left the caching configuration of one Seattle point of presence reverted while the routing configuration in front of it was not.

   Root cause

   Dependency failure

   Component

   CDN and edge networkObject storage

   Contributing factor

   Partial rolloutOverlapping failuresSignal buried in noise

   Detection

   Users filed reports in the PyPI support tracker; one report isolated the 502s to a single Fastly cache node using the x-served-by headers of the failing responses.

   Remediation

   • Fastly's network operations removed the routing override that sent part of PyPI's traffic to the affected point of presence
   • Fixed the B2-to-S3 fallback for the case where B2 does not respond at all (infra#237)
   • Exempted suffix and multi-range requests from segmented caching (infra#241, infra#243)
   • Fastly patched the canary configuration bug and removed all PSF traffic from its canary cohort
   • Moved the segmented-caching exemption check after URL normalization (infra#245)

   Action items

   • Improve the logging configuration of the file hosting service (infra#238)
   • Reject HTTP methods that a file host should not receive (infra#239)
   • Follow a support ticket with Fastly about segmented-caching range handling, and revert PyPI's own handling code once Fastly fixes it
   • Return to Fastly's canary program once clearer controls and notifications exist around that traffic
   • Hire an infrastructure engineer, partly for PyPI, to catch conditions like this earlier

   Lessons

   None of the PyPI configuration bugs were new; they had been in the configuration for a while and came out only when real traffic on large files with range requests forced an investigation. The 5xx baseline was already noisy, so the start of the Fastly problem was easy to miss, and part of that noise turned out to be one of the bugs.

   View case

By root cause2