Incident Library
Applied filters ComponentCloud load balancerFactorSingle point of failureDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
    Symptoms

    At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

    Service outageHigh error rate
    Trigger

    The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

    Root cause
    Cloud provider outage
    Component
    CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules
    Contributing factor
    Single point of failure
    Detection

    Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

    Remediation
    • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
    • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
    • Created a regional WAF for the ALB with the security team
    • Created a new IngressClass with the ACM certificate, the WAF and access log settings
    • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing
    Action items
    • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
    • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
    • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB
    Lessons

    VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

    View case
Check your cluster for this

This case involves CDN and edge network and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules
Contributing factor
Single point of failure
Organisation
ANDPAD
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026