Cases7 cases

1 to 5 of 7

1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs

   Python Software Foundation2026

   Symptoms

   For about two weeks in August 2026, some PyPI users got intermittent 502 and 503 errors when downloading files from files.pythonhosted.org, and installations from PyPI failed. The first two reports of persistent 502s were opened in the support tracker on 17 August.

   High error rate

   Trigger

   A partial rollback during a canary deployment inside Fastly's network, whose canary cohort had included PyPI's traffic for a number of years, left the caching configuration of one Seattle point of presence reverted while the routing configuration in front of it was not.

   Root cause

   Dependency failure

   Component

   CDN and edge networkObject storage

   Contributing factor

   Partial rolloutOverlapping failuresSignal buried in noise

   Detection

   Users filed reports in the PyPI support tracker; one report isolated the 502s to a single Fastly cache node using the x-served-by headers of the failing responses.

   Remediation

   • Fastly's network operations removed the routing override that sent part of PyPI's traffic to the affected point of presence
   • Fixed the B2-to-S3 fallback for the case where B2 does not respond at all (infra#237)
   • Exempted suffix and multi-range requests from segmented caching (infra#241, infra#243)
   • Fastly patched the canary configuration bug and removed all PSF traffic from its canary cohort
   • Moved the segmented-caching exemption check after URL normalization (infra#245)

   Action items

   • Improve the logging configuration of the file hosting service (infra#238)
   • Reject HTTP methods that a file host should not receive (infra#239)
   • Follow a support ticket with Fastly about segmented-caching range handling, and revert PyPI's own handling code once Fastly fixes it
   • Return to Fastly's canary program once clearer controls and notifications exist around that traffic
   • Hire an infrastructure engineer, partly for PyPI, to catch conditions like this earlier

   Lessons

   None of the PyPI configuration bugs were new; they had been in the configuration for a while and came out only when real traffic on large files with range requests forced an investigation. The 5xx baseline was already noisy, so the start of the Fastly problem was easy to miss, and part of that noise turned out to be one of the bugs.

   View case
2. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

   ANDPAD2026

   Symptoms

   At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

   Service outageHigh error rate

   Trigger

   The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules

   Contributing factor

   Single point of failure

   Detection

   Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

   Remediation

   • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
   • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
   • Created a regional WAF for the ALB with the security team
   • Created a new IngressClass with the ACM certificate, the WAF and access log settings
   • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing

   Action items

   • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
   • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
   • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB

   Lessons

   VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

   View case
3. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case
4. IL-0357 Up to 22% of Vercel static file requests failed when part of its global cache was lost during the AWS us-east-1 outage

   Vercel2025

   Symptoms

   On 20 October 2025 at 07:45 UTC, while Vercel was moving traffic away from its us-east-1 region (iad1) during the AWS us-east-1 outage, a cascading failure took out a percentage of Vercel's global caching infrastructure. Failures of static file serving peaked at about 22% of traffic, the largest impact of the whole event, until service was restored at 08:18 UTC.

   High error rate

   Trigger

   The AWS us-east-1 region (Vercel's iad1) failed on 20 October 2025, and Vercel began re-routing traffic and function invocations away from it; at 07:45 UTC a cascading failure followed in the global caching infrastructure.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkObject storage

   Contributing factor

   –

   Detection

   The first automated alerts, for errors across Vercel services, fired at 06:55 UTC at the start of the us-east-1 disruption, 50 minutes before the cache failure began; incidents were created automatically and on-call engineers were paged.

   Remediation

   • Re-routed end-user traffic away from iad1 into neighbouring data centers
   • Re-routed function invocations to configured backup or secondary regions
   • Restored static file serving at 08:18 UTC and mitigated the root cause of the cascading cache failure

   Action items

   • Write a detailed technical postmortem leading to systematic mitigation of the failure modes these disruptions exposed
   • Continue and intensify rehearsals of major regional disruptions

   View case
5. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024

   Sentry2024

   Symptoms

   On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

   High error rateService outageAutoscaler misbehaviour

   Trigger

   In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

   Root cause

   Capacity exhaustion

   Component

   CDN and edge networkAutomated capacity manager

   Contributing factor

   Automation amplificationExcessive blast radius

   Action items

   • Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
   • Change the autoscaling rules to use stronger signals than CPU load
   • Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
   • Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity

   View case

By root cause6