Cases9 cases

1 to 5 of 9

1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

   ANDPAD2026

   Symptoms

   At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

   Service outageHigh error rate

   Trigger

   The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules

   Contributing factor

   Single point of failure

   Detection

   Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

   Remediation

   • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
   • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
   • Created a regional WAF for the ALB with the security team
   • Created a new IngressClass with the ACM certificate, the WAF and access log settings
   • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing

   Action items

   • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
   • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
   • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB

   Lessons

   VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

   View case
2. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications

   UK Ministry of Justice2026

   Symptoms

   At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

   High error rateService outage

   Trigger

   The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerWeb application firewall rules

   Contributing factor

   Unrepresentative staging

   Detection

   At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

   Remediation

   • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
   • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
   • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
   • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00

   Action items

   • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
   • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
   • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
   • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress

   Lessons

   An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

   View case
3. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
4. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
5. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case

By root cause8