Incident Library

IL-0438 ANDPAD · 2026-07-16awsnetworkingkubernetes

Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

VPC Origins lets CloudFront deliver content from applications hosted in private VPC subnets, with an ALB, an NLB or EC2 instances as the origin. Builder Work runs as containers on EKS, the AWS Load Balancer Controller turns Kubernetes Ingress resources into ALBs, and the surrounding AWS infrastructure is managed with Terraform. Builder Work had used a domain-sharding layout with separate front-end and API domains, which caused CORS errors in browsers and 401 errors during Cognito authentication because the CloudFront domain and the Host header the ALB received did not match, and the Ingress rules written to work around them were hard to operate. To remove these problems the ALB was moved from a public subnet to a private one and used as CloudFront's VPC origin, so outside traffic came only through CloudFront. Traffic between CloudFront and the origin therefore passed through an AWS-managed shared subsystem. On 16 July 2026, according to the AWS Health notice, a capacity constraint in the internal shared subsystem that manages VPC Origins connections broke that path from about 16:45 to about 20:20 JST, while CloudFront as a whole kept working.

VPC Origins was the only origin configured, so ANDPAD decided to route traffic temporarily to an internet-facing ALB without CloudFront. The internet-facing Ingress and ALB had been deleted when VPC Origins was introduced, so there was nothing simple to roll back to and the Ingress had to be created again. The existing WAF was a global WAF for CloudFront, managed in the us-east-1 global scope, which cannot be attached to a regional ALB, so a regional WAF for the ALB was created with the security team. A new IngressClass combined the ACM certificate, the WAF and access log settings, and Ingresses for normal traffic, the admin screens (Cognito authentication with VPN access restriction) and redirects were added. Builder Work's deployments are managed by Argo Rollouts, so the new Ingress also had to be attached to the Rollout resource and promoted, which made the switch take considerable time. Builder Work recovered once requests were served through this layout. The SRE team handled the infrastructure and the Builder Work development team checked the application.

Severity
–
Scope
Builder Work was almost entirely unreachable; other ANDPAD services showed no problems after login
Users affected
–
Region
–
Time to detect
15m
Time to resolve
–

Symptoms

At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

CloudFront itself kept working; only the VPC Origins feature failed, and distributions using it returned many 504 Gateway Time-out errors. The SRE team found that among ANDPAD's products only one CloudFront distribution, Builder Work's, used VPC Origins. Screens and functions after login in the other ANDPAD services showed no problems.

The failure lasted about 3.5 hours, from about 16:45 to about 20:20 JST. Builder Work's ALB sat in a private subnet and was reached only through CloudFront VPC Origins, so the only options during the failure were to wait for AWS to recover or to switch the ALB back to internet-facing.

Symptom Service outageHigh error rate

Root cause

Root cause Cloud provider outage

Component CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules
Contributing factor Single point of failure
Trigger
The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

Detection

Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

Remediation

Action items

Lessons

VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network, Cloud load balancer, Ingress controller among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.