Cases4 cases

1 to 4 of 4

1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs

   Python Software Foundation2026

   Symptoms

   For about two weeks in August 2026, some PyPI users got intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, and installations from PyPI failed. The first two reports of persistent 502s were opened in the support tracker on 17 August.

   High error rate

   Trigger

   A partial rollback during a canary deployment inside Fastly's network, whose canary cohort had included PyPI's traffic for a number of years, left the caching configuration of one Seattle point of presence reverted while the routing configuration in front of it was not.

   Root cause

   Dependency failure

   Component

   CDN and edge networkObject storage

   Contributing factor

   Partial rolloutOverlapping failuresSignal buried in noise

   Detection

   Users filed reports in the PyPI support tracker; one report isolated the 502s to a single Fastly cache node using the x-served-by headers of the failing responses.

   Remediation

   • Fastly's network operations removed the routing override that sent part of PyPI's traffic to the affected point of presence
   • Fixed the B2-to-S3 fallback for the case where B2 does not respond at all (infra#237)
   • Exempted suffix and multi-range requests from segmented caching (infra#241, infra#243)
   • Fastly patched the canary configuration bug and removed all PSF traffic from its canary cohort
   • Moved the segmented-caching exemption check after URL normalization (infra#245)

   Action items

   • Improve the logging configuration of the file hosting service (infra#238)
   • Reject HTTP methods that a file host should not receive (infra#239)
   • Follow a support ticket with Fastly about segmented-caching range handling, and revert PyPI's own handling code once Fastly fixes it
   • Return to Fastly's canary program once clearer controls and notifications exist around that traffic
   • Hire an infrastructure engineer, partly for PyPI, to catch conditions like this earlier

   Lessons

   None of the PyPI configuration bugs were new; they had been in the configuration for a while and came out only when real traffic on large files with range requests forced an investigation. The 5xx baseline was already noisy, so the start of the Fastly problem was easy to miss, and part of that noise turned out to be one of the bugs.

   View case
2. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task

   Canva2024

   Symptoms

   On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

   Service outageOOMKilledHigh latencyPacket loss

   Trigger

   At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

   Root cause

   Dependency failure

   Component

   CDN and edge networkAPI gatewayAmazon ECS

   Contributing factor

   Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug

   Remediation

   • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
   • Redirected canva.com to the status page
   • Restored traffic gradually, starting with Australian users under strict rate limits

   Action items

   • Write a runbook for rerouting, blocking and progressively restoring traffic
   • Show users a more informative error page when canva.com is unavailable
   • Raise the baseline number of API Gateway tasks and the memory of each task
   • Add load shedding rules to the API Gateway for similar traffic patterns
   • Load test the API Gateway regularly
   • Deploy the patch for the thread-locking bug in the telemetry library
   • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
   • Use page load completion events as a second canary indicator for page asset releases
   • Experiment with adding another rollout stage to the canary to give more time to detect issues
   • Add timeouts so user requests do not wait excessively long for assets
   • Work with Cloudflare to understand the system interactions behind the incident

   View case
3. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

   Wikimedia Foundation2023

   Symptoms

   On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

   Network unreachable

   Trigger

   Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingCDN and edge network

   Contributing factor

   Single point of failureIgnored early signal

   Detection

   Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

   Remediation

   • Depooled eqsin from DNS
   • Repooled eqsin

   Action items

   • Create a backup GRE tunnel for eqsin (T327265)
   • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
   • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)

   Lessons

   The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

   View case
4. IL-0398 GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN

   Government Digital Service2021

   Symptoms

   On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.

   Service outage

   Root cause

   Dependency failure

   Component

   CDN and edge network

   Contributing factor

   –

   Detection

   The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.

   Remediation

   • Waited 15 minutes to judge the primary CDN outage, then decided to fail over to the secondary CDN
   • Deployed the prepared DNS record changes, and traffic began to route to the secondary CDN within 30 minutes
   • Switched back to the primary CDN after it recovered, and monitored for an hour before declaring the incident resolved

   Action items

   • Hold a formal incident review of what happened and how the team responded
   • Consider how to improve the speed and transparency of communications
   • Consider how planned changes to platform technology and processes can make recovery from future incidents faster

   Lessons

   The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.

   View case

By root cause1