Cases3 cases

1 to 3 of 3

1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

   ANDPAD2026

   Symptoms

   At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

   Service outageHigh error rate

   Trigger

   The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules

   Contributing factor

   Single point of failure

   Detection

   Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

   Remediation

   • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
   • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
   • Created a regional WAF for the ALB with the security team
   • Created a new IngressClass with the ACM certificate, the WAF and access log settings
   • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing

   Action items

   • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
   • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
   • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB

   Lessons

   VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

   View case
2. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone

   UK Ministry of Justice2022

   Symptoms

   On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.

   High latencyIntermittent timeouts

   Trigger

   Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerIngress controller

   Contributing factor

   –

   Detection

   Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.

   Remediation

   • AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone

   Action items

   • Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository

   View case
3. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA

   loveholidays2019

   Symptoms

   On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

   Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour

   Trigger

   A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

   Root cause

   Cloud provider outage

   Component

   GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler

   Contributing factor

   Unrepresentative stagingOverlapping failuresIgnored early signal

   Detection

   The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

   Remediation

   • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
   • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
   • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
   • used kubectl port-forward in place of the broken internal Ingresses
   • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back

   Action items

   • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
   • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
   • Re-evaluate termination grace periods that were set to arbitrary large values
   • Read the GKE release notes before every cluster upgrade
   • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
   • Keep environments as identical as possible, including resource names

   Lessons

   Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

   View case

By root cause1