Cases3 cases

1 to 3 of 3

1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

   ANDPAD2026

   Symptoms

   At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

   Service outageHigh error rate

   Trigger

   The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules

   Contributing factor

   Single point of failure

   Detection

   Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

   Remediation

   • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
   • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
   • Created a regional WAF for the ALB with the security team
   • Created a new IngressClass with the ACM certificate, the WAF and access log settings
   • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing

   Action items

   • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
   • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
   • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB

   Lessons

   VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

   View case
2. IL-0357 Up to 22% of Vercel static file requests failed when part of its global cache was lost during the AWS us-east-1 outage

   Vercel2025

   Symptoms

   On 20 October 2025 at 07:45 UTC, while Vercel was moving traffic away from its us-east-1 region (iad1) during the AWS us-east-1 outage, a cascading failure took out a percentage of Vercel's global caching infrastructure. Failures of static file serving peaked at about 22% of traffic, the largest impact of the whole event, until service was restored at 08:18 UTC.

   High error rate

   Trigger

   The AWS us-east-1 region (Vercel's iad1) failed on 20 October 2025, and Vercel began re-routing traffic and function invocations away from it; at 07:45 UTC a cascading failure followed in the global caching infrastructure.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkObject storage

   Contributing factor

   –

   Detection

   The first automated alerts, for errors across Vercel services, fired at 06:55 UTC at the start of the us-east-1 disruption, 50 minutes before the cache failure began; incidents were created automatically and on-call engineers were paged.

   Remediation

   • Re-routed end-user traffic away from iad1 into neighbouring data centers
   • Re-routed function invocations to configured backup or secondary regions
   • Restored static file serving at 08:18 UTC and mitigated the root cause of the cascading cache failure

   Action items

   • Write a detailed technical postmortem leading to systematic mitigation of the failure modes these disruptions exposed
   • Continue and intensify rehearsals of major regional disruptions

   View case
3. IL-0584 Cloud CDN at Osaka served corrupted JavaScript files, and a single-page app failed with a SyntaxError in western Japan

   Monicle2024

   Symptoms

   Some users reported that a Monicle web service could not be used. The service is a single-page app (SPA) served through Google Cloud Load Balancing and Cloud CDN, and in the affected browsers the app stopped with a SyntaxError. Users in eastern Japan were not affected; users in western Japan were.

   Response quality degradation

   Root cause

   Cloud provider outage

   Component

   CDN and edge network

   Contributing factor

   –

   Detection

   Some users reported that they could not use the system, and a war room was opened right away because of the product and the extent of the impact.

   Remediation

   • Purged the Cloud CDN cache after the corruption was seen at KIX (Osaka) but not at NRT (Tokyo)
   • Enabled compression_mode for Cloud CDN content on Cloud Load Balancing, after which the app worked for users again

   Action items

   • Stay aware of which parts of the service are single points of failure, remove them where the cost is justified, and prepare to respond quickly to failures in PaaS and SaaS products

   Lessons

   The author states that there is no specific countermeasure for this incident yet. The lesson drawn is to keep in mind which parts of the service are single points of failure and to be ready to act quickly on failures in PaaS and SaaS products.

   View case

By root cause1