Cases1 case

1 to 1 of 1

1. IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deploy

   ITANDI2020

   Symptoms

   Right after a deploy to the production ECS (Fargate) service, HTTPCode_ELB_5XX_Count on the service's ALB jumped. The service receives about 10,000 requests a minute, and roughly 20% of them got a 5XX.

   High error rate

   Trigger

   A deploy recreated the ECS tasks, and at that moment communication between the ALB and its backends became unstable.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerAmazon ECS

   Contributing factor

   –

   Detection

   The team saw HTTPCode_ELB_5XX_Count jump on the ALB right after the deploy, together with target connection errors.

   Remediation

   • Restarted the ECS tasks, which did not help
   • Moved the ALB's ap-northeast-1d subnet to another subnet in the same zone, which did not help
   • Moved the six ECS tasks running in 1d to 1a and 1c, which did not help
   • Removed the 1d subnet from the ALB, which did not help
   • Changed the ALB idle timeout, which did not help
   • Rolled the ECS tasks back to the version before the deploy, which did not help
   • Created a new ALB with the same VPC, subnets, backends and security groups
   • Attached both ALBs to the ECS service
   • Sent one tenth of the traffic to the new ALB with a weighted DNS record
   • Moved all traffic to the new ALB after it returned no 5XX

   Lessons

   When a graph shows something that should be impossible on the network, suspect the AWS side as well.

   View case

By root cause1